Post on 01-Dec-2021
transcript
Fund Methodology Paper
Prosperity Fund – Evaluation and Learning ii
Table of Contents
1. Introduction .................................................................................................................. 1
Introduction ............................................................................................................. 1 Purpose and Scope of the Paper ............................................................................ 1
2. Evaluation Architecture ............................................................................................... 2
3. Methodology ................................................................................................................ 3
The Evaluation Matrix ............................................................................................. 3 Selection of Evaluation Methodology ...................................................................... 4 Evaluation Methodology ......................................................................................... 5 Synthesis Process .................................................................................................. 9 Data Sources and Triangulation .............................................................................. 9
4. Evaluation Cycle and Products ................................................................................ 10
The Annual Evaluation Cycle ................................................................................ 10 Sequencing of Evaluation Activities ...................................................................... 11 Annual Evaluation Activities .................................................................................. 15
5. Learning ..................................................................................................................... 16
6. Evaluation Principles ................................................................................................. 18
7. Risks and mitigation measures ................................................................................ 19
8. Annexes...................................................................................................................... 21
Annex 1: Evaluation Matrix .......................................................................................... 21 Annex 2: Evaluation Methodologies ............................................................................. 33
List of Tables
Table 1: Sequencing of evaluation questions ......................... Error! Bookmark not defined. Table 2: Risks and mitigation measures .............................................................................. 19
List of Figures
Figure 1: Evaluation Architecture .......................................................................................... 2 Figure 2: Six step contribution analysis ................................................................................. 8 Figure 3: Evaluation design steps ......................................................................................... 5 Figure 4: The E&L evaluation cycle ..................................................................................... 11 Figure 5: Year 1 evaluation activities ................................................................................... 15 Figure 6: Evaluation learning cycle...................................................................................... 17
Fund Methodology Paper
Prosperity Fund – Evaluation and Learning iii
Version History
Fund Methodology Paper
WYGEL5-INC-11-05 Revision 3
Revision Written By Checked By Approved By
Revision 0 Ann Bartholomew Paddy Abbot, Michael Schultz PFMO
Revision 1 Ann Bartholomew Paddy Abbot, Michael Schultz PFMO
Revision 2 Ann Bartholomew Paddy Abbot, Michael Schultz PFMO
Revision 3 Ann Bartholomew Paddy Abbot, Michael Schultz PFMO
This report was prepared during the inception phase of the Prosperity Fund Evaluation and Learning contract. It forms an annex to the main Inception Report.
It was produced and approved by the Prosperity Fund Management Office before the main Inception Report and Workplan were finalised and agreed.
If there is any inconsistency between this annex and the main Inception Report and Workplan, the main Inception Report and Workplan provides the agreed position.
Fund Methodology Paper
Prosperity Fund – Evaluation and Learning iv
Table of Abbreviations
E & L Evaluation & Learning
EQs Evaluation Questions
G&I Gender & Inclusion
HMG Her Majesty’s Government
ICAI Independent Commission for Aid Impact
IDA International Development Act
IDC International Development Committee
IP Implementing Partner
IPA Infrastructure and Projects Authority
KII Key Informant Interview
MDB Multilateral Development Bank
M&R Monitoring & Reporting
MREL Monitoring, Reporting, Evaluation & Learning team of the PFMO
NAO National Audit Office
ODA Overseas Development Assistance
OECD-DAC
Organisation for Economic Cooperation & Development – Development Assistance Committee
PD Project Director
PF Prosperity Fund
PFMO Prosperity Fund Management Office
PIDG Private Infrastructure Development Group
PM Project Manager
QA Quality Assurance
Q&A Questions and Answers
SRO Senior Responsible Officer
TA Technical Assistance
TAG Technical Advisory Group
ToC Theory of Change
ToR Terms of Reference
VfM Value for Money
Fund Methodology Paper
Prosperity Fund – Evaluation and Learning 1
1. Introduction
Introduction
The Evaluation and Learning (E&L) service to the Prosperity Fund (PF) is provided by three
firms, led by WYG, together with Integrity and LTS. We work alongside the Monitoring
and Reporting (M&R) service provider, PA Consulting, and their partner, The Economist
Intelligence Unit.
The E&L approach places the usefulness of the evaluation to stakeholders at the centre of
evaluation design and implementation. For this reason, over our seven-month inception
period, we have engaged with PF stakeholders to enable us to design our E&L approach and
methodology, as well as establish relationships that will support implementation.
The purpose of the E&L Function is to generate lessons that can be used to enhance the
effectiveness of the PF at project, programme, programme ‘families’ and Fund
levels. Evaluations will respond to Programme Teams’ learning and information needs:
providing evaluation evidence they need to do their job better. We will also be evaluating what
has been achieved, how and why.
E&L works closely alongside M&R. While the focus of the Fund’s M&R systems is on
accountability (i.e. identifying and explaining PF spending and results achievement); E&L
processes focus principally on learning. The roles are nevertheless related, with data and
findings from M&R expected to feed into programme and fund level evaluations. For this
reason, the E&L and M&R service providers have and will continue to closely coordinate our
delivery and service designs.
Purpose and Scope of the Paper
This paper outlines the evaluation methodology that will be used to assess the fund’s overall
performance in terms of accountability and progress towards results and provide learning to
improve performance. The main purposes of the PF Evaluation are as follows:
• Learning about performance - what works, why, how and in what contexts – to inform
design, future Fund strategy, plans and actions to improve performance
• Accountability – to ensure efficient and well-targeted spending that is delivering value
for money and contributing to the achievement of primary purpose benefits (ODA
requirement) and secondary benefits
There will also be a strong user focus of the evaluation, as it is intended that the primary users
of the fund evaluation will be programme managers and Senior Responsible Officers (SROs),
and their implementing partners, the PFMO and management and governance structures of
the fund (Portfolio Board and Ministerial Board). Other users will be the implementing
departments, other development partners, the International Development Committee (IDC),
the Independent Commission for Aid Impact (ICAI) Infrastructure and Projects Authority (IPA),
Multilateral Development Banks (MDBs) and the private sector.
This paper draws on discussions with the PFMO MREL team, the Monitoring and Reporting
contractor (M&R) and discussions held internally in the Evaluation and Learning (E & L) team
on Fund-level and Programme evaluation. It also builds on the E & L Synthesis Strategy, the
Fund Methodology Paper
Prosperity Fund – Evaluation and Learning 2
Gender Inclusion Strategy, the Learning Strategy and Plan and the terms of reference (TOR)
for Programme Evaluations.
The paper outlines the overall evaluation architecture in section 2, the evaluation methodology
in section 3 and the evaluation cycle and products to be delivered in section 4. Section 5
addresses learning within the evaluation, section 6 outlines the evaluation principles and
section 7 outlines potential risks and their mitigation. The evaluation matrix can be found in
Annex 1, the pros and cons of different methodologies considered by the team, as well as a
description of methodologies chosen is in Annex 2.
2. Evaluation Architecture
The evaluation architecture builds on the PF Evaluation Framework (see E & L Evaluation
Framework, February 2018), creating an interlinked structure for the overall evaluation and for
how the evaluation can be delivered to satisfy the various users. This structure is described in
Figure 1 below, which indicates the 4 levels of the evaluation.
Figure 1: Evaluation Architecture
The 4 levels relate to programme evaluations, family-level evaluations, thematic evaluations
and the fund evaluation:
• Programme Level Evaluations: will be developed together with the respective
Programme Managers, who are effectively commissioning the evaluations with
guidance from the E&L Portfolio Lead (and other E&L team members deemed
Fund Methodology Paper
Prosperity Fund – Evaluation and Learning 3
necessary to advise). This will involve the selection of particular projects for evaluation
or particular aspects of several projects, within a programme (e.g. testing particular
assumptions, or projects in a particular geography, or particular project types)1.
• Family Level Evaluation: will be commissioned by the Fund Evaluation, the PFMO or
suggested by programme managers and will explore facets of the ‘families’ of
programmes. These are groupings of projects or programmes that share similar
outcomes and can be grouped per (Fund level) intermediate outcome (see E&L
Synthesis Strategy, March 2018 for a detailed description of families). Evidence from
programme evaluations and other available sources and case-studies will be used to
evaluate specific aspects of the family’s contribution to each intermediate outcome
(which are part of primary purpose) and secondary benefits. Whilst commissioned from
the Fund level, case studies that feed into the family level evaluations will be
undertaken as part of programme evaluations so as to retain ownership of evaluative
activities by Programme Managers at programme level and ensure learning benefits
are generated at both programme and at Family/Fund level.
• Thematic evaluations: will be commissioned by the Fund Evaluation and agreed with
the PFMO and are evaluations that cut across the fund, which may be large macro-
evaluations or smaller focused studies and will synthesise evidence from questions
asked at programme level (necessitating clear integration into programme
evaluations). They are likely to require case studies at project or programme level,
necessitating engagement through the programme managers. They will be designed
at Fund level with necessary engagement at programme level in selection of sample
cases. Examples of thematic evaluations could be gender and social inclusion, VfM,
or adaptive learning across the fund.
• Fund level evaluation: draws on the evaluative and monitoring evidence generated by
the evidence emerging from the thematic, family level and programme evaluations in
each annual cycle to provide evaluative learning and implications for the fund
management, design, processes and governance, as well as assessing fund progress
and performance.
3. Methodology
The Evaluation Matrix
The evaluation methodology is based on the theory of change (ToC) as its organising
structure, as applying a theory-based evaluation approach allows the testing of assumptions
underlying the causal chain from inputs to intermediate outcomes, outcomes and contributions
towards impact (See E&L Evaluation Framework, February 2018). The process of developing
the evaluation questions (EQs) involved mapping these to the ToC, to provide a framework
1 More on the scope and objective of these programme level evaluations is described in the Programme TOR submitted February 2018.
Fund Methodology Paper
Prosperity Fund – Evaluation and Learning 4
for analysis that allows questions relating to fund performance and learning to be answered,
as well as the ToC to be tested.
The questions to be answered and at what level of the evaluation this will be undertaken, will
be guided by an evaluation matrix (see Annex 1) which outlines the following in relation to
each of the EQs.
• EQs and sub-questions
• Evaluative activities and level of evaluation
• Information sources
• Responsibility for information and data collection
This provides an overview of how E&L will address each evaluation question in terms of data
requirements, methodology and who is responsible. At the same time, the questions and sub-
questions will cascade down to programme and intervention levels to ensure that a consistent
set of questions will be answered in each evaluation at programme level (See Programme
ToR submitted 12 February 2018). The data and analysis from this will in turn be synthesized
to provide information at the fund level to respond to the EQs and test the ToC.
Selection of Evaluation Methodology
The selection of evaluation methodology will depend on the purpose of the evaluation
(accountability or learning), the EQs that we are answering and the relative importance of each
question. This informs the overall evaluation approach, for example to use either a utilisation-
focused and/or theory-based technique. The methodologies selected (i.e. contribution
analysis, process tracing, outcome mapping) then inform the data collection, analytical and
synthesis methods and tools for each EQ. Whether there are any other areas of interest
beyond just looking at the EQs will be explored with the programme managers and the PFMO
that may also affect the choice of methodologies.
By taking into consideration the type of EQ we can identify which methodologies are best
suited to answering the EQ. Questions such as ‘how much’, ‘what’, ‘how’ and ‘why’ require
different methodologies, as statistical approaches are good at answering ‘how much’
questions but not ‘how’ and ‘why’. Contribution analysis approaches are good at answering
‘what’, ‘how’ and ‘why’, whereas realist evaluation approaches are particularly good at
exploring ‘how’ and ‘why’ questions.
The features of the intervention will also guide choices on methodology, as for some
interventions, the attributes of the intervention may make it difficult or impossible to use some
methodologies, so we will conduct evaluability assessments to guide methodological choices.
The steps in evaluation design that we will follow are illustrated in Figure 3.
Fund Methodology Paper
Prosperity Fund – Evaluation and Learning 5
Evaluation design
Evaluation Methodology
The evaluation will use a number of methodologies according to which EQs are being
answered, with the selection of methodologies based on which is considered by the team to
be most appropriate. Table 1, in Annex 2 indicates the types of methodologies identified by
the evaluation team as realistic options for addressing each of the EQs. The main methods
that will be used are realist evaluation, contribution analysis (CA), process tracing and
qualitative comparative analysis. Some methods such as using random control trials or quasi-
experimental methods have been ruled out as they are not considered appropriate for the
attributes of the Fund, the requirements of the evaluation or the context within which the Fund
operates.2 As part of this process, the pros and cons of the different methodologies in terms
2 The proposed methods were included based on a literature review of common methods applied to a range of complex, multi-country programmes and funds and guided by best practice as described
1. Approach
The overall approach that ensures the evaluation answers the EQs. The evaluation purpose and EQs inform the approach.
E.g. theory-based evaluation, utilisation-focused evaluation, etc.
The approach influences the choice of methodology
2. Methodology
How we understand the EQ (e.g. type of causality etc.) informs the evaluation process/methodology to answer the EQ.
(Methodology may be viewed as a type or 'sub-set' of approach)
E.g. contribution analysis, realist evaluation, process tracing, etc.
The methodology influences how we collect, analyse and synthesise the data
3. Analytical and synthesis methods
What we do to analyse and synthesise data
Analytical methods - e.g. context analysis, barrier analysis, etc.
Analytical products - e.g. case studies, scorecards, etc.
Synthesis methods - e.g. thematic analysis, family synthesis, etc.
4. Data collection methods and tools
a) What methods we use to collect data.
E.g. interviews, observations, FGDs; document review; survey; etc.
b) What tools we use to collect data
E.g. interview protocols/ questionnaires/scorecards
1. Theory-based evaluation 2. Contribution analysis 3. Case studies
4a) Interviews and document review
4b) Semi-structured interview
protocols/scorecards
Purpose: accountability and learning
Example EQ: What difference did the programme make and how?
Figure 2: Evaluation design steps
Fund Methodology Paper
Prosperity Fund – Evaluation and Learning 6
of their appropriateness for the PF evaluation were considered by the team. These are outlined
in Table 2 in Annex 2. A description of the techniques chosen is given in Box 1 below.
in Choosing Appropriate Evaluation Methods: A Tool for Assessment and Selection, October 2016. Published by Bond, Society Building, 8 All Saints Street, London N1 9RL, UK
Fund Methodology Paper
Prosperity Fund – Evaluation and Learning 7
Table 1: Main evaluation methodologies
Methodology Brief description
Contribution analysis The notion of a ‘contributory’ cause, recognizes that effects are produced by several causes at the same time, none of which might be necessary nor sufficient for impact. A development intervention along with other factors produce an impact, as part of a causal package; but they need not and sometimes the impact is ‘caused’ by a quite different mix of causes in which the intervention plays no part. The causal package is sufficient but can also be unnecessary: i.e. there are other ‘paths’ to impact, which may or may not include the intervention. The intervention is a contributory cause of the impact if: the causal package with the intervention was sufficient to bring about the impact, and the intervention was a necessary part of that causal package.
See Mayne, J. (2008). Contribution Analysis: an approach to exploring cause and effect. ILAC Brief 16. Institutional Learning and Change (ILAC) Initiative (CGIAR).
Realist evaluation Realist approaches assume that nothing works everywhere or for everyone, and that context really does make a difference to programme outcomes3. Consequently, policy-makers and practitioners need to understand how and why programmes work and don’t work in different contexts, so that they are better equipped to make decisions about which programmes or policies to use and how to adapt them to local contexts.
Realist evaluation asks not ‘what works?’ or ‘does this programme work?’ but asks instead ‘what works for whom in what circumstances and in what respects, and how?’ A realist question contains some or all of the elements of ‘how and why does this work and/or not work, for whom, to what extent, in what respects, in what circumstances and over what duration?’
See Gill Westhorp (2014) Realist Evaluation: An Introduction. Methods Lab Overseas Development Institute London
Process tracing Process tracing dissects causation through causal mechanisms between the observed variables, primarily in case studies. In essence, the focus of process tracing is on establishing the causal mechanism, by examining the fit of a theory to the intervening causal steps. Theorists using process tracing ask ‘how does “X” produce a series of conditions that come together in some way (or do not) to produce “Y”?’ By emphasising that the causal process leads to certain outcomes, process tracing lends itself to validating theoretical predictions and hypotheses.
Befani, B. and J. Mayne (2014) Process Tracing and Contribution Analysis: A Combined Approach to Generative Causal Inference for Impact Evaluation
QCA Qualitative comparative analysis (QCA) is an approach to systematic cross-case comparison. It establishes what factors, common across cases, can explain similar outcomes; or what factors could explain different outcomes. Unlike most methods intended to draw generalised lessons across cases, QCA does not look at variables in isolation. It focuses on combinations or configurations of factors within single cases; and allows generalisation only to the extent that these holistic combinations are preserved.
QCA is appropriate to identify the preconditions and make sense of the diversity in results across small numbers of cases when there are several but not many causal factors. It is not appropriate when the explanation is only one case.
See Befani, B. (2013). Multiple Pathways to Policy Impact: Testing an Uptake Theory with QCA. CDI Practice Paper 5. Institute of Development Studies
The core evaluation method that will be used in the evaluation is CA. This will be important
particularly for the synthesis methodology (see E & L Synthesis Strategy, March 2018). CA
Fund Methodology Paper
Prosperity Fund – Evaluation and Learning 8
refers to a theory-based approach that aims to confirm that an intervention is a contributory
cause to a given outcome. It is used to assess cause and effect relationships in circumstances
when impacts result from a complex interplay of actions by multiple players, and a variety of
contextual factors4. To apply contribution analysis, the evaluation team will follow six iterative
steps (Figure 3) which use the theory of ToC to guide the analytical process5.
The first step involves identifying the attribution problem(s), or evaluation EQs, while the
second step involves elaborating the theory of change, which are both steps that have been
completed during the inception phase. As a result, we have identified what is likely to be the
contribution of the PF programmes to the achievement of primary benefits and identified key
contexts where the PF is likely to have an influence and the necessary conditions for
contributing to changes. Steps 3-6 will be undertaken in the implementation phase and are
iterative, identifying hypotheses on contribution claims to be explored during subsequent
iterations and evidence gaps to be addressed. Step 3 involves review of existing data related
to key contexts where PF is likely to contribute to changes through existing case-studies,
portfolio analysis and secondary data. The contribution from PF will be identified and the role
of other contributing factors. Step 4 focuses on the different sources of data assembled at
each step of the causal pathway in the Fund ToC. Where a change story is strong, the team
will engage stakeholders to provide feedback on the contribution evidence. In step 5 additional
evidence will be collected where there are gaps in the evidence or stakeholders express
considerable uncertainty over the contribution story, leading to step 6 where the contribution
story is revised and strengthened to generate lessons and insights for the PF.
Figure 3: Six step contribution analysis
4 Mayne J. (2008) ‘Contribution Analysis: An Approach to Exploring Cause and Effect’, Institutional Learning and Change (ILAC) Initiative, ILAC Brief 16.
5 Mayne, J. (2012). Contribution analysis: coming of age? Evaluation vol. 18 no. 3, pp. 270-280.
Fund Methodology Paper
Prosperity Fund – Evaluation and Learning 9
Synthesis Process
As implementation and results generation will largely happen through projects undertaken by
the programmes, a systematic process for structuring, drawing together, analysing and
presenting the evaluation findings against the over-arching framework will be used (See E&L
Synthesis Strategy, March 2018). This synthesis approach will support evaluation at three of
the four levels of the evaluation as follows:
• Family-level synthesis evaluations will explore contribution to primary purpose and
secondary benefit at intermediate outcome level. Groupings that align to an
intermediate outcome and the synthesis of programme evaluation evidence by a
‘family synthesis’. will synthesise evidence from the programme evaluations and other
available sources (e.g. learning from evaluations of similar programmes) and case
studies that evaluate specific aspects of the family’s contribution to each intermediate
outcome.
• Thematic evaluations will assess important factors that affect the achievement of
results, but which cut across the causal pathways of the Fund ToC. These will use
synthesis from existing programme and family-level evaluation together with specific
case studies to gather additional information and learning.
• The Fund evaluation will draw together information from programme, family and
thematic level evaluations, MR data and from external research or literature data to
make assessments of overall performance and contribution to the Fund ToC.
Data Sources and Triangulation
Data will be drawn from secondary data sources as outlined in the E&L Report on Secondary
Data Quality and Availability (submitted by E&L March 2018). The MR indicators will be a key
source of information for E&L activities with all the EQs needing to draw on data encapsulated
in Prospero and in particular the contextual, portfolio management, fund performance and
output indicators. Programme level sources will be important and include documentation
outlining the programme design, approval and business case stages and programme delivery
stages. The latter will include quarterly progress reports, annual reviews and programme
completion reviews, while financial reporting will be useful for information on budgets and
expenditure. Individual project document will be drawn on such as inception reports, quarterly
progress reports, annual reports and project completion reviews. This information will also be
supplemented by reputable international data sources in relation to different sectors and
families, as well as key informant interviews and the use of scorecards and rubrics to gather
opinions from key stakeholders identified for each option6.
A clear process has been developed for triangulating data sources with four sources of data
triangulation as follows:
6 It is intended that the use of scorecards will cover EQs related to primary purpose, Secondary benefit, VfM, Gender and possibly sustainability and contribution to learning.
Fund Methodology Paper
Prosperity Fund – Evaluation and Learning 10
1. Triangulation of various secondary and primary data sources in the production of thematic and
programme evaluation reports and Fund-level analysis;
2. Triangulation of findings within families by the evaluation teams;
3. Validation of family-level findings with programme teams, incorporating their inputs;
4. Meta-analysis of evaluation products at the Fund-level.
Our approach to triangulation is further outlined in Annex 2 of the E & L Synthesis Strategy,
March 2018.
4. Evaluation Cycle and Products
The Annual Evaluation Cycle
The E&L team will follow an annual evaluation cycle with 5 stages, which will guide the
development and implementation of activities and outputs (see Figure 4). This will focus on
the selection of an annual suite of evaluations as the first stage, with design as the second
which will be undertaken as described in section 3 above. This will be a collaborative process
with the main output an approach paper for each evaluation. Stage 3 will be the
implementation phase with a series of evaluations undertaken collecting data and information
and validating findings and will involve analysis and sense making, as well as data synthesis
and report writing. Evaluation products from this stage will be:
i) Programme Evaluation Reports; ii) Family Synthesis Report iii) Thematic Evaluation Reports iv) Fund Annual Evaluation Report
Fund Methodology Paper
Prosperity Fund – Evaluation and Learning 11
Figure 4: The E&L evaluation cycle
The final stage 5 will involve a review of performance and learning, a review of the ToC and
development of priorities for the next cycle. It should be noted that learning will not just take
place in stage 5, as there will be learning touch points throughout the cycle as described in
section 5 on learning below. Evaluation products from this stage will be:
i) Annual Fund Level Theory of Change Review
Beyond the main evaluation products (i.e. approach papers and annual evaluation products)
the programme evaluations would be expected to deliver interim products and learning outputs
that support the learning and information needs of the PMs and the synthesis evaluations.
These interim products would be detailed in the annual programme approach paper. More
detail on the fund level evaluation products is outlined in Box 3 below.
Sequencing of Evaluation Activities
The focus in the early years of the evaluation will be formative, with more summative
evaluation being undertaken in the later years, when programmes are being fully implemented
and are achieving identifiable results. In year 1 activities will primarily be focused on creating
a baseline and establishing the evidence base, with years 2 to 4 following an established
Fund Methodology Paper
Prosperity Fund – Evaluation and Learning 12
annual evaluation cycle7. There will also be an emphasis on learning in the early years to allow
real time learning to improve performance, with assessing performance and results becoming
more prominent in the later stages of the evaluation. Therefore, the focus on specific EQs or
emphasis on addressing different parts of EQs will change over time. This is shown in Table
1 below and indicates that there is unlikely to be robust evidence available on results until the
end line, given the short-time period for implementation.
7 In year 1 some activities will provide information for the HMG Spending Review expected in 2019 .e.g. an assessment of extent to which the cross-cutting factors (3-pillar understanding of sustainability, G&I, VfM) is incorporated into design and a baseline assessment of likely contribution (i.e. contribution claims) across the Fund.
Table 2: Sequencing of evaluation questions
Evaluation Question Years 2-3 Assumptions Year 4 Assumptions
What has been or is likely to be achieved as a result of the PF?
1. What evidence is there that the Prosperity Fund is likely to contribute to the intended outputs and intermediate outcomes in the ToC, as well as any unintended or unexpected effects?
Some evidence through annual ToC review to assess possible contributions.
Dependent on progress in programme tendering & implementation
Answered8 Evidence on intermediate outcomes is available (may not be due to short time period)
2. Which types of interventions, sectors and country settings have been more and less successful in contributing to the achievement of primary benefits?
Limited evidence n/a Answered May not be able to comment on all sectors and country settings, depending on programme progress
3. Which types of interventions, sectors and country settings have been more and less successful in contributing to the achievement of secondary benefits?
Limited evidence n/a Answered May not be able to comment on all sectors and country settings, depending on programme progress
4. What evidence is there that Prosperity Fund interventions will be sustainable and ensure environmental sustainability, will be self-financing and lead to inclusive growth that reduces inequality?
Limited evidence Thematic cross-cutting review should provide evidence on appropriateness of design
Partially answered (fully answered through impact evaluation)
Judgement on sustainability likely, but unlikely on link to gender and inclusive growth
What factors have contributed to or hindered these achievements?
5. What factors have contributed to the achievement of primary benefits and secondary benefits?
Limited evidence n/a Answered Extent to which this is addressed again depends on programme progress
6. How has the balance and relationship between primary and secondary benefits across the
Limited evidence Some evidence likely from assessing
Answered See assumptions for EQ 2 & 3
8 ‘Answered’ means that the questions will be addressed within the boundaries of available information and assuming programme implementation has happened as planned.
Evaluation Question Years 2-3 Assumptions Year 4 Assumptions
portfolio influenced the achievement of results? programme design and results frameworks
7. Which assumptions and the causal pathways outlined in the ToC remain valid, which have been adapted and what refinements need to be made?
Answered on an annual basis
Dependent on progress in programme tendering & implementation
Answered Assuming there is robust evidence on all pathways. May not occur if some programmes delayed
8. To what extent is the institutional governance set-up of the Prosperity Fund more or less effective in achieving i) primary benefits; ii) secondary benefits; iii) other results?
Some initial evidence likely
Some evidence likely from assessing programme design and results frameworks
Answered See assumptions for 2 & 3
9. What types of approaches, governance and management arrangements have been more and less effective for achieving results and demonstrate good approaches to supporting inclusive growth and VfM?
Little robust evidence likely, although some may emerge on VfM
Dependent on the extent to which VfM systems have been established by programmes
Answered for VfM
Inclusive growth will be addressed in impact assessment
Dependent on the extent to which VfM systems have been established by programmes
10. To what extent have the Prosperity Fund interventions contributed to results that support gender equality, women's economic empowerment and social inclusion in line with the UK’s Gender Equality Act and the Prosperity Fund Policy and Guidance and the Prosperity Fund Gender and Inclusion Framework?
Some initial evidence likely
Assuming sufficient information from the thematic evaluation
Answered Gender disaggregated data is available and there is data collected on social inclusion
What can be learned from the PF experience to date to improve ongoing and future programming?
11. How is the Prosperity Fund learning and why is action on this learning happening more and less successfully?
Evidence available Dependent on progress in programme tendering & implementation
Answered Cooperation on learning activities by programmes
12. Which Prosperity Fund lessons in translating outputs into intermediate outcomes are sufficiently robust for wider learning?
Limited evidence Answered Evidence on intermediate outcomes available
Fund Methodology Paper
Prosperity Fund – Evaluation and Learning 15
Annual Evaluation Activities
In year 1, the annual cycle will be slightly different as it will focus on establishing a baseline
for measuring performance, reviewing the ToC. The activities that are envisaged to be
undertaken by the evaluation team are outlined in Figure 5.
Figure 5: Year 1 evaluation activities
In terms of the Programme Level Evaluations, there will be an evaluation of all programmes
each year, although in year 1 all programmes will not be covered as not all will be fully
operational and at a point where a review would be useful. It is expected that around 15
programmes evaluations will take place in year 1. Where programme evaluations will take
place, activities will focus on agreement on EQs, conducting baseline assessments and sense
making of findings with Programme Managers. It is anticipated that all programmes will have
an evaluation from years 2 to 4. It is important to note that not all projects within a programme
will be included in this process and inclusion will depend on the focus of the evaluation and
the implementation progress of individual projects, while the number will be restricted by the
available budget.
For the Family Evaluations the baseline work from the programme evaluations will be used to
build causal pathways and context mechanisms assumptions of proposed families in year 1.
At an early stage the family groupings will be tested and reviewed within programme teams.
Family synthesis will be undertaken on all the families, which draws from the programme
Fund Methodology Paper
Prosperity Fund – Evaluation and Learning 16
evaluations relevant to each family, and there will also be case study synthesis (See E & L
Synthesis Strategy, March 2018).
Thematic evaluations will draw on evidence from the suite of E&L Programme Evaluations,
Family Evaluations and comparative analysis from other comparable funds or programmes.
Thematic studies may also use case-studies to explore specific issues or test specific
assumptions/hypothesis from the secondary evidence base. Where possible, case-studies for
thematic evaluations will be planned in combination with programme evaluations, to minimize
evaluation fatigue, but also to ensure ‘local’ relevance of the wider evaluation findings.
Themes will be chosen based on an analysis of critical issues generated by the Annual Fund
Evaluation and the ToC Review or from programme managers if common themes for enquiry
are identified. Thematic evaluations in year 1 will consist of cross-cutting assessments of how
selected PF elements have been incorporated into programme design and processes that are
in place to monitor them. Gender and inclusion, VfM, secondary benefits and sustainability will
be the four areas focused on in this work. There will be some smaller cross-cutting studies
that will involve piloting methodology and scorecards for these cross-cutting issues. This
means there will be 3 major thematic studies started in year 1 (but not completing until Year
2 Q1), and 5 smaller early win studies.
From year 2 onwards there will be a series of large and small thematic studies going forward.
The Fund evaluation will focus on baseline activities in year 1 to develop a baseline for the
EQs and the ToC Review. This will include synthesis analysis of the evaluations and synthesis
at the other 3 levels of the evaluation. Sense making of baseline findings will take place and
an annual reflection workshop for the Fund ToC. Evidence of progress and results will be
generated for the Spending Review in 2019. Going forward the Fund Evaluation will continue
to generate evidence against the EQs and undertake an Annual Fund Level ToC Review and
produce an Annual Fund Evaluation Report.
The Annual Fund Evaluation Report will assess the core EQs against the three pillars that the
PMFO will report against: VfM, primary purpose and secondary benefits. It will draw on the
evaluative and monitoring evidence generated throughout each annual cycle and provide
evaluative learning and implications for the fund on management, design, process and
governance, as well as assessing fund performance and contribution to achieving primary and
secondary benefits. The overall assessment of fund progress and performance will be
disaggregated by programme, project type family and theme, to understand which
interventions are performing well or not well, how and why and what contextual factors are
influencing performance. The ToC Review will test assumptions and causal pathways, using
the evidence generated by the suite of E&L evaluations and M&R monitoring data to provide
an interactive assessment of how new evidence either confirms or challenges existing ToC
assumptions and causal pathways (See Proposed Revisions to and Methodology for Future
Revisions to the Prosperity Fund Theory of Change March 2018). This will include an
assessment of plausibility of evidence, including monitoring evidence and generate
recommendations on aspects of monitoring and reporting that will result in evidence-based
revisions to the ToC.
5. Learning
Learning for the Fund Evaluation will follow the process developed in the E&L Learning
Strategy and Plan (March 2018). This emphasises the value of learning throughout the
Fund Methodology Paper
Prosperity Fund – Evaluation and Learning 17
evaluation cycle and stakeholder participation in design, analysis and follow-up to making
evaluations useful and underpins the E&L Team’s annual evaluation and learning cycle
(Figure 6).
This cycle will drive evidence-based review of progress (on what has been achieved, how and
why) as reflected in the Thematic Evaluations, Annual Theory of Change Review and Annual
Report. This will facilitate the sharing of, reflection on, and use of the evaluation evidence to
support decision making at Fund levels. process which responds to PFMOs and PMs learning
needs through the following processes9:
• Validate EQs / Plans: to ensure relevance to own learning needs and opportunities
• Evaluation Stock Take: to document their own knowledge related to the programme /
fund alongside previous evaluation / research findings
• Contribute Evidence / Know How: to collaboratively build the evidence base to answer
EQs
• Validate Findings and Develop Recommendations: to ensure preliminary evaluation
findings are robust and that evaluation recommendations are relevant
9 From E&L Learning Strategy and Plan, submitted February 2018.
Figure 6: Evaluation learning cycle
Fund Methodology Paper
Prosperity Fund – Evaluation and Learning 18
• Make sense of it: to put evaluation findings and recommendations back into the current
programme context and use them as a critical perspective to review assumptions and
causal pathways
• Plan Follow-Up Actions: to take responsibility for action on learning gains as part of
active / adaptive management
• Wider Learning / Enquiry: to ensure that E&L plans for sharing of externally valid
lessons from the evaluation or for commissioning further evaluations to address gaps
in knowledge are appropriate to own learning needs and opportunities
6. Evaluation Principles
The following principles will guide our overall approach to evaluation:
• Usefulness: findings will be relevant and high quality to be perceived as useful and
geared to current operational concerns of programme managers and the PFMO.
• Independence: For the evaluation to be impartial, it will be free from bias in findings,
analysis and conclusions. This implies independence from the those implementing the
programme, so we will ensure that the involvement of stakeholders does not
compromise objectivity and that there is transparency in the roles played.
• Credibility: the evaluation must be perceived as objective, rigorous and impartial, with
credibility ensured by the professional quality of evaluators and the rigor of methods.
• Accuracy: we will ensure that data and findings are accurate and robust which will be
validated through a rigorous process of sampling, analysing, interpreting and
triangulating data and findings.
• Transparency: credibility and usefulness will also depend on the transparency of the
evaluation and we will ensure that findings are easily available to all stakeholders.
• Do no harm: Interventions and activities will be based on ‘do no harm’ principles and
incorporate ethical practices.
• Confidentiality: we will ensure confidentiality of information, privacy and anonymity of
study participants. We will communicate clearly to prospective participants any limits
to confidentiality.
• Respect cultural sensitivities: we will take account of differences in culture, local
behavior and norms, religious beliefs and practices, sexual orientation, gender roles,
disability, age and ethnicity and other social differences such as class when planning
evaluations and communicating findings.
• Sensitivity towards gender and social inclusion: The evaluation will incorporate gender
and social inclusion-sensitive considerations.
• Tailored learning approach: this will be based on a clear understanding of the different
stakeholders, including beneficiaries and their differing monitoring, evaluation and
learning needs. Findings will be fed back in inclusive ways.
• A robust multi-layered QA process will be applied to ensure the generation of high
quality evidence. This will ensure the evaluation is governed by explicit principles
Fund Methodology Paper
Prosperity Fund – Evaluation and Learning 19
covering: inputs (methodologies, people, resources); processes (oversight and
governance, management arrangements, stakeholder engagement, communications);
and deliverables (outputs).
7. Risks and mitigation measures
There are potential risks and limitations that may result from the Fund evaluation methodology.
The key operational risks are outlined in the table below with probability, impact and actions
that will be taken by the team to mitigate these risks10.
Table 3: Risks and mitigation measures
Risk Mitigation Measures Probability Impact
1. Risk of bias in programme monitoring and reporting as the system is based on self-reporting
Mitigated by E&L programme evaluations and thematic reviews which will verify data
Medium High
2. EQs may not be able to be addressed due to data limitations i.e. disaggregation by sex, age and socioeconomic group
Any data limitations will be highlighted in the baselining process and/or the annual report to allow systems to be established
to collect data or to adapt the questions
Medium Medium
3. Annual Evaluation cycles are not aligned with programme reporting cycles
Close coordination with Programme Managers by Family Evaluation Leads to ensure annual cycles fit as closely as possible to those of programmes
Medium Medium
4. Delays in programme implementation may mean few results emerge before the end line or beyond this point
The development of a baseline allows the further evaluation of performance after 2022 or for an impact evaluation in 2025
Medium High
5. M&R fund level indicators are too high level to allow analysis of PF contribution to achieving indicators
E&L will work with M&R to ensure that indicators are appropriate and match to the ToC, which will be reviewed annual in
the ToC review
High High
6. Complexity and uncertainty over programme pathways
The Annual ToC Review will update the ToC and review casual pathways and assumptions
High High
7. Programmes decline to be involved in evaluation activities
A dispute mechanism will be established to mediate between E&L and programme managers. E&L products will be tailored to provide useful lessons to programme managers.
Medium Medium
8. The high number of evaluation activities causes a heavy burden on Programme Managers
There will be close coordination between the members of the core evaluation team to ensure that all activities are
Medium High
10 See E & L PF Risk Register for a full list of operational, financial and contractual risks
Fund Methodology Paper
Prosperity Fund – Evaluation and Learning 20
Risk Mitigation Measures Probability Impact
coordinated without duplication of evaluation activities
9. E & L Evaluation teams are unable to deliver high quality inputs
E&L retains a highly experienced core team, augmented by internationally recognised experts and partners. Regular communication with the team and PFMO will ensure that evaluation products are in line with PFMO's expectations and relevant external QA standards.
Low High
10. Lack of agreement between E&L and programmes on appropriate approach
E & L evaluation approach is based on a core offering of four evaluation types and is flexible to adapt to context of individual programmes. Early baseline activities are with PMs to establish learning need and to interpret/tailor EL's programme evaluation TORs for each programme.
Medium Medium
Pro
sp
erity
Fu
nd
– E
va
lua
tion
an
d L
ea
rnin
g 2
1
8. Annexes
Annex 1: Evaluation Matrix
Evaluation Questions Indicative Sub Evaluation Questions Evaluative Activities Information Sources Responsibility
EQ1: What evidence is there that the Prosperity Fund is likely to contribute to the intended outputs and intermediate outcomes in the ToC, as well as any unintended or unexpected effects?
1.1 What have been the observed or likely outputs, outcomes and results from the Fund's implementation? (Note that expected results may change over the duration of the Fund and may be greater in number / different to those initially outlined in the Business Case. Expected outputs and outcomes should be identified through consultation with all Fund stakeholders)
1.2 What other (unintended or unexpected) outputs and outcomes have arisen or might be logically expected to arise from the
Fund's implementation?
1.3 What are the posited causal pathways linking the Fund interventions to these outputs / outcomes? Was the PF a contributory cause to any outputs/outcomes along the pathways to impact?
1.4 How much evidence supports / refutes these posited pathways?
1.5 What alternative theories might explain observed outputs / outcomes (e.g. contextual factors, other interventions, beneficiary actions, etc.)
1.6 What role did the Fund play in bringing about the observed outcomes/outputs and
Fund Level Evaluation which will bring together evidence results from across the range of evaluation activities to understand progress, including annual ToC reviews and updates based on evidence.
Family level Evaluations of families to assess contribution to achievement of intermediate outcomes in the Fund ToC. This will collect units of evidence support/refuting purported causal
pathways resulting from
evidence collected from various evaluations during implementation
Programme evaluations commissioned by Programme managers that focus on contribution to results and reviews of programme ToCs and causal pathways
Thematic evaluations that focus on collecting evidence of changes at various levels of ToC, supporting understanding of contribution to those
M&R Indicators including output and outcome indicators in M&R fund level indicators and particularly output and outcome indicators listed in programme
and project logframes
Fund, programme and project documentation analysis (ToC
analysis)
Other sources of information on the ToC, e.g. results of ToC workshops (where undertaken) and/or key informant interviews (KIIs)
Evidence resulting from all E&L primary data collection / analysis exercises (e.g. literature reviews, surveys, focus groups, etc.) as relevant.
M&R (Core Indicators)
Programme leads & project IPs (programme and project level results indicators)
E&L leads (i.e. technical lead, portfolio lead,
thematic lead)
Pro
sp
erity
Fu
nd
– E
va
lua
tion
an
d L
ea
rnin
g 2
2
Evaluation Questions Indicative Sub Evaluation Questions Evaluative Activities Information Sources Responsibility
did it act as a trigger or support factor to bring about the changes observed?
changes and evidence of alternative explanations for causes of change
EQ2: Which types of interventions, sectors and country settings have been more and less successful in contributing to the achievement of primary benefits?
2.1 Are there any meaningful patterns in the way that PF results (especially outputs and intermediate outcomes as identified under EQ1) are distributed by sector, country and activity which suggest that some of these programmes / countries / sectors / activities may have been associated with an environment more / less conducive to Fund success?
2.2 What contextual factors do key stakeholders and beneficiaries suggest have enabled or prevented projects from contributing to the achievement of primary outcomes and intermediate outcomes? How do these compare across programmes / countries / sectors / activities?
2.3 What other evidence is there (e.g. from programme and project reporting) of success factors / barriers and are there any emerging patterns?
Year 1 Programme Evaluations analyse the respective programme ToCs and provide baseline data on primary benefits, disaggregated by social inclusion and context.
Typology of interventions and contexts developed after Year 1 Programme Evaluations (and updated annually), to be built into evaluation sampling for Portfolio Evaluations and at the fund level to understand trends.
Fund level Evaluation bringing together evidence at the fund level in the annual report to understand success factors and barriers to performance across different programmes
Evaluations of families, sampling interventions on basis of typology, establishes causal pathways and interventions’ contributions to change through use of case-based methods for the policy-change interventions (TA, capacity building, policy change) and quasi-experimental or other impact evaluations for the other interventions. To also include contextual analysis (allows comparison between contexts).
M&R Indicators provide results data on primary benefits / programme level outcomes. (NB: disaggregation by sex, age and socioeconomic group is not included in the MR data or programme documents.)
Programme level reporting and information including MR indicators and other results indicators (TBC), and programme level data such as sector, contextual information, intervention type, internal programme processes and capacity mapped into Fund level analysis matrix
Critical success factors' and barriers to project/programme implementation and achievement, identified by programmes and E&L
Description of different processes (and/or ToCs) of different interventions, sectors and activities and mechanisms of
M&R (Core Indicators)
Programme leads & project IPs (programme and project level results indicators; critical success factors; lessons learned)
E&L Leads (evidence of results, evidence of critical success factors and
barriers)
Pro
sp
erity
Fu
nd
– E
va
lua
tion
an
d L
ea
rnin
g 2
3
Evaluation Questions Indicative Sub Evaluation Questions Evaluative Activities Information Sources Responsibility
Programme Evaluations insofar as they include programme-level analyses of contribution to primary outcomes and contextual analysis of factors influencing programme implementation (more detailed assessment at programme level).
change and the benefits mapped against the ToC
Information on contextual factors influencing programme implementation (e.g. programme/institutional, contextual issues such as legislation, issues related to beneficiaries themselves) correlating with causal pathways identified as credible / evidence-based against a pre-defined rating system
Information from Programme Managers via Learning Groups structured per family and other learning activities
EQ3: Which types of interventions, sectors and country settings have been more and less successful in contributing to the achievement of secondary benefits?
3.1 Are there any meaningful patterns in the way that PF secondary benefits created through in-country partnerships are distributed by sector, country and activity ?
3.2 If so, do these patterns reveal which PF programmes / host countries / sectors / activities have been more or less successful in creating secondary benefits?
Year 1 Programme Evaluations analyse the respective programme ToCs and provide baseline data on secondary benefits.
Typology of interventions and contexts developed after Year 1 Programme Evaluations (and updated annually), to be built into evaluation sampling for Portfolio Evaluations and at the fund level to understand trends.
Evaluations of families, sampling interventions on basis of typology, establishes causal pathways
M&R Indicators provide results data on secondary benefits / programme level secondary outcome results.
Programme level reporting and information including MR indicators and other results indicators (TBC), and programme level data such as sector, contextual information, intervention type, internal programme processes and capacity mapped into Fund level analysis matrix
M&R (Core Indicators)
Programme leads & project IPs (programme and project level results indicators; critical success factors; lessons learned)
Pro
sp
erity
Fu
nd
– E
va
lua
tion
an
d L
ea
rnin
g 2
4
Evaluation Questions Indicative Sub Evaluation Questions Evaluative Activities Information Sources Responsibility
and interventions’ contributions to generating secondary benefits through use of case-based methods for the policy-change interventions (TA, capacity building, policy change) and quasi-experimental or other impact evaluations for the other interventions. To also include contextual analysis (allows comparison between contexts).
Programme Evaluations insofar as they include programme-level analyses of contribution to secondary outcomes and contextual analysis of factors influencing achievement (more detailed assessment at programme level).
Thematic Evaluation focusing on aspects of how PF programmes contribute to different secondary
benefits across the portfolio of interventions
Critical success factors' and barriers to project/programme implementation and achievement, identified by programmes and E&L
Description of different processes (and/or ToCs) of different interventions, sectors and activities and mechanisms of change and the benefits mapped against the ToC
Information on contextual factors influencing programme implementation (e.g. programme/institutional, contextual issues such as legislation, issues related to beneficiaries themselves) correlating with causal pathways identified as credible / evidence-based against a pre-defined rating system
Information from Programme Managers via Learning Groups structured per family and other learning activities
E&L Leads (evidence of results, evidence of critical success factors and
barriers)
EQ4: What evidence is there that the changes
4.1 Is there evidence that the benefits generated by the PF programmes and
M&R Indicators provide results data on environmental
M&R (Some indicators could be
Pro
sp
erity
Fu
nd
– E
va
lua
tion
an
d L
ea
rnin
g 2
5
Evaluation Questions Indicative Sub Evaluation Questions Evaluative Activities Information Sources Responsibility
supported by the Prosperity Fund interventions will be sustainable and ensure environmental sustainability, will be self-financing and lead to inclusive growth that reduces inequality?
projects will last beyond their implementation?
4.2 In particular, is there evidence that any of the PF activities have led to in-country public or private sector financing commitments? What are the barriers (if any) to self-financing?
4.3 To what extent have project inputs been environmentally sustainable?
4.4 What, if any, outputs and outcomes can be said to contribute to environmental sustainability, and how?
4.5 Is there evidence that PF activities and outputs have produced benefits (or will do so in the future) that are broad based, reach poorer quintiles, address key gender barriers and harness opportunities for women’s economic participation
Year 1 Programme Evaluations establish baseline data against which social inclusion can be measured and articulate pathways for reducing inequality / which beneficiaries are targeted and how.
Evaluations of Families that focus on how different families of programmes articulate and can measure against pathways for reducing inequality.
Programme evaluations information to support answering these questions will likely be collected at mid-term and final evaluation stage.
Thematic Evaluations for example, that explore likely environment impact / sustainability of programmes and the management efforts to generate self-financing following PF funding and the sustainability of inputs, how the PFMO is managing integration of these issues into programmes, and how programmes consider inequality within their programmes.
Annual Fund Report which includes a discussion on emerging results and beneficiary analysis and a meta-analysis of fund and programme data, including comparative analysis between programmes.
sustainability or inclusive growth (for example, # beneficiaries by income level at start of programme, per programme
Evidence of sustainability of benefits
# and % of projects / project activities or outputs with mechanisms for self-financing post-PF (e.g. female/male-owned businesses with self-financing mechanisms)
Ways in which/ number of projects have utilised inputs in an environmentally-sustainable manner and contributed to environmental sustainability
Instruments (incl. VfM scorecard) to assess gender and social inclusion
Synthesis to allow comparison of achievement of theories of change within similar interventions for gender and inclusive growth and sustainability
Validation with the Programme Managers via Learning Groups
commissioned for collection)
E&L (all other data collection and analysis), Programme Leads, Thematic Lead, Portfolio Lead
Pro
sp
erity
Fu
nd
– E
va
lua
tion
an
d L
ea
rnin
g 2
6
Evaluation Questions Indicative Sub Evaluation Questions Evaluative Activities Information Sources Responsibility
EQ5: What factors have contributed to the achievement of primary and secondary benefits?
5.1 When looking at the ToC (Fund and programme levels), what factors (i.e. assumptions) are expected to enable the achievement of primary and secondary benefits (this may include factors such as the way programmes / projects are implemented)? Are these realistic and evidence-based?
5.2 When considering the causal pathways which are the most credible / evidence-based factors (inputs, context, scope, mechanisms) that appear to have contributed to primary and secondary benefits?
5.3 Are there any meaningful patterns in the way that primary and secondary benefits (as identified under EQ1) are distributed which suggest that certain characteristics of the programmes / countries / sectors / activities producing these benefits may have been associated with an environment more / less conducive to achievements?
5.4 What factors do key stakeholders and beneficiaries suggest have enabled or prevented interventions/projects from contributing to the achievement of primary and secondary benefits?
Synthesis of EQs 2 & 3 will identify patterns in families or kinds of interventions that are more or less successful in achieving primary and secondary benefits.
Evaluations of Families establish through case studies the factors establishing the effects of the programmes, referring to and drawing on contextual factors affecting achievement of results and a meta-analysis of the broad factors or barriers influencing achievement of primary and secondary results.
Programme Evaluations to look at what contributes to the achievement of primary and secondary benefits within the context of each programme evaluated.
Annual Fund Evaluation will bring together themes and results exploring trends in different factors that influence achievement of primary and secondary results, will include annual ToC review and contribution analysis of PF contribution to any changes observed
M&R Indicators provide results data on primary and secondary benefits.
Instruments developed for portfolio evaluation family toolkits to systematically identify and classify contextual factors
Validation with the Programme Managers via Learning Groups
structured per family
Synthesis against the family and Fund ToCs
Views and perspectives of PF stakeholders on the primary and secondary benefits and factors affecting achievement of results
Literature review on key factors emerging from PF data
Programme evaluation and thematic study findings on primary and secondary benefits and factors affecting them
M&R (primary and secondary benefit indicators)
E&L (additional primary and secondary benefit indicators, plus analysis of causal pathways), Technical Lead, Portfolio Lead and Thematic Lead
Pro
sp
erity
Fu
nd
– E
va
lua
tion
an
d L
ea
rnin
g 2
7
Evaluation Questions Indicative Sub Evaluation Questions Evaluative Activities Information Sources Responsibility
ToC review findings (adaptations, evidence supporting posited assumptions, etc.)
EQ6: How has the balance and relationship between primary and secondary outcomes across the portfolio influenced the
achievement of results?
6.1 What is the balance between a focus on primary and secondary benefits in terms of:
- Fund level reporting and communications;
- HMG and other key stakeholder perspective on (and influence over) the Fund
- Programme prioritisation
- Programme activities
- Programme beneficiaries
- Expected results (at programme level e.g. in logframes)
6.2 Are primary benefits and beneficiaries and secondary benefits and beneficiaries adequately, discreetly and clearly defined in programme design?
6.3 Overall (i.e. at an aggregate Fund level) what are the primary and secondary benefits, e.g. in terms of men and women expected to be reached, men and women reached, etc.? How do these compare / balance out?
6.4 How do PF stakeholders foresee (if at all) that primary and secondary benefits will be
Draws on material from EQs 2, 3 & 5.
Thematic evaluations, for example, focussing on management decisions of sampled programmes and on the manner in which the outcomes and their explanatory factors have been derived from the design and management decisions.
Fund level evaluation which relies on a Year 1 baseline set establishing the balance of benefits and tracking changes to this every year
Evaluations of Families evaluations focusing on the synthesis of the interdependence between
primary and secondary outcomes
Other indicators on primary and secondary benefits tracked and
mapped
Perspectives of PF stakeholders on primary and secondary benefits, how they interact and their relative importance
Mentions of primary and secondary benefits in Fund and programme documentation (e.g. business cases, meeting minutes, press releases, etc.)
# programmes / projects / activities / etc. rated (by E&L) has having lesser or greater focus on primary or secondary benefits
Primary and secondary benefit indicators listed in programme and project logframes.
E&L only (Technical Lead, Portfolio Lead and Thematic Lead)
Pro
sp
erity
Fu
nd
– E
va
lua
tion
an
d L
ea
rnin
g 2
8
Evaluation Questions Indicative Sub Evaluation Questions Evaluative Activities Information Sources Responsibility
mutually supportive? Is this plausible? Has it happened in practice?
EQ7: Which assumptions and the causal pathways outlined in the ToC remain valid, which have been adapted and what refinements need
to be made?
7.1 In practice, how have assumptions guided / directed programme and Fund implementation?
7.2 Which assumptions as outlined in the PF theory of change and PF programme theory of change have held true/not held true?
7.3 Which assumptions may have facilitated the Fund's success(es) and which may have
hindered it?
7.4 Have and how have assumptions been adapted and refined as the Fund and programmes have been implemented? Has this refinement affected (improved?) implementation? If so, how?
7.5 Are there any additional refinements that could / should be made or new assumptions emerging?
Programme Evaluations to include initial structured analysis of the programme ToCs (down to the individual project level) and assumptions that are then reviewed every year.
Evaluations of families to include initial structured analysis of the ToCs within each family, surfacing the assumptions stated and implicit.
Fund level Evaluation includes a review of evidence emerging annually against the ToC, including assumption and causal pathway testing and contribution analysis and other methods to
test causality
Meta-synthesis of responses to EQs 2-5 against the family and Fund ToCs, leading to revisions
of the ToCs on an annual basis
Validation with the Programme Managers via Learning Groups structured per family and with the
PFMO
Views and perspectives of PF stakeholders on the underlying ToC, including the context and underlying assumptions
Data on intervention contexts, implementation, and results from e.g.
- programme reporting and
monitoring;
- data collection undertaken as part of programme evaluation activities or commissioned for thematic studies;
- literature review / context mapping and experts in the field.
E&L only (Technical Lead, Portfolio Lead and Thematic
Lead)
EQ8: To what extent is the institutional governance set-up of the Prosperity Fund
8.1 What are the different processes underpinning the Prosperity Fund's governance and management at Fund level (e.g. coordination, decision-making
Draws on material from EQs 2-3
M&R Indicators on governance and management effectiveness / efficiency.
M&R (primary and secondary benefit indicators + data on governance and
Pro
sp
erity
Fu
nd
– E
va
lua
tion
an
d L
ea
rnin
g 2
9
Evaluation Questions Indicative Sub Evaluation Questions Evaluative Activities Information Sources Responsibility
more or less effective in achieving i) primary benefits; ii) secondary benefits; iii) other results?
processes, monitoring, financial management, communications, guidance and support, etc.)?
8.2 What are the different processes underpinning the Prosperity Fund's governance and management at programme and project level (as above)?
8.3 Through what kinds of delivery set-up (e.g. FCO, DFID, MDB) have Prosperity Fund programmes and projects been
delivered?
8.4 Is there any patterning between the results achieved under these different models of governance and delivery at programme and project level (note that this question may already have been answered under EQs 2, 3 and 5)?
8. 5 In analysing the likely factors affecting outputs, outcomes and impacts, which governance and delivery mechanisms appear to have been more or less successful in supporting the achievement of results?
8.6 Has the PF institutional and cross government governance, management and learning processes contributed to results at the HMG level in terms of capacity or influence of FCO or capacity and skills in
cross governmental working?
Review of Programme and Fund Documentation describing Fund, programme and project governance, delivery procedures, set-up and timelines.
Evaluations of families, that for example focus on the VfM of different families’ causal pathways to achieving results.
Thematic Evaluations, for example looking at the management decisions of sampled programmes and the manner in which the outcomes and their explanatory factors have been derived from the institutional governance.
Fund level evaluation which will rely on a Year 1 baseline assessment of processes and compare with other relevant funds. Evidence emerging will be reviewed annually to understand how processes are adapting and contributing to results. This will include VfM analysis on the effectiveness of the processes, as well as comparative analysis, systems mapping, contribution analysis and ToC analysis.
Programme evaluations that capture information on programme level implantation and governance set up and how this has contributed to performance.
Programme documentation (describing Fund, programme and project governance and delivery procedures and set-up, as well as results reporting).
Other relevant funds and comparison documentation
Portfolio and thematic studies focus on institutional governance, level of influence
Views and perspectives of PF stakeholders on the primary and secondary benefits and factors
affecting them.
Programme evaluation, portfolio and thematic study findings on primary and secondary benefits and factors affecting them.
ToC review findings (adaptations, evidence supporting posited
assumptions, etc.).
management effectiveness)
E&L (additional primary and secondary benefit indicators, plus analysis of governance arrangements) (Technical Lead, Portfolio Lead and Thematic Lead)
Pro
sp
erity
Fu
nd
– E
va
lua
tion
an
d L
ea
rnin
g 3
0
Evaluation Questions Indicative Sub Evaluation Questions Evaluative Activities Information Sources Responsibility
EQ 9: What types of programmes, approaches and governance and management arrangements have been more and less effective for achieving results and demonstrate good approaches to supporting inclusive
growth and VfM?
11.1 What are and how have different programmes, approaches and governance/management arrangements resulted in inclusive growth and VfM (note that the scope of 'inclusive growth' and 'VfM' as "results" will be defined and programmes and projects mapped accordingly)?
11.2 Which types of programmes, approaches and governance/management arrangements have been more and less successful in achieving this inclusive growth and VfM?
11.3 On the basis of this (and other) analysis, which programmes, approaches and governance and management arrangements can be said to demonstrate good practice for supporting inclusive growth
and VfM?
Draws on material from EQs 4-6, 8-9.
Evaluations of Families to understand how different families of programmes achieve VfM and inclusive growth.
Programme Evaluations that explore inclusive growth and VfM at a programme level.
Thematic Study on process e.g. addressing PF data on VfM systems and requirements (indicators, procurement, timeliness and cost-effectiveness of outputs, strategies and measures in place to enhance delivery and mitigate risk)
Fund level evaluation synthesising evidence of meaningful patterns in how inclusive growth and
VfM is addressed across programmes
M&R Indicators on inclusive growth and VfM
Other relevant indicators collected in logframes
Description of programme approaches, governance and management arrangements for measuring and using VfM data
# times programmes changed approach based on VFM evidence (based on surveys/management meeting
minutes)
Evidence of good practice approaches to inclusive growth and VfM from other programmes (benchmarking)
E&L and M&R based on their VfM approach
EQ10: To what extent have the Prosperity Fund interventions contributed to results that support gender equality, women's economic empowerment and social inclusion in line with the UK’s Gender Equality Act and the Prosperity Fund Policy
9.1 To what extent the programme has put in place gender/inclusion-responsive Fund and programme mechanisms that meet minimum compliance with GEA (undertaken due diligence and do no harm) to understand the gender and distributional impact of interventions?
9.2 The extent to which the programmes support opportunities for/removing barriers to women’s economic empowerment, for
Thematic Evaluation on Gender, drawing on programme documentation, contextual/country analysis on gender and social inclusion issues
(including policy and legal frameworks)
Fund level evaluation will rely on a Year 1 baseline that assess the extent to which gender has been integrated into programmes. A meta-analysis of evidence emerging to support
M&R indicators disaggregated by gender
Instruments to assess gender developed for programme and
portfolio evaluation toolkits
M&R (disaggregation of indicators)
E&L (Technical Lead, Portfolio Lead and Thematic
Lead)
Pro
sp
erity
Fu
nd
– E
va
lua
tion
an
d L
ea
rnin
g 3
1
Evaluation Questions Indicative Sub Evaluation Questions Evaluative Activities Information Sources Responsibility
and Guidance and the Prosperity Fund Gender and Inclusion Framework?
benefits to flow to the poor and excluded groups/for SMEs/informal sector?
9.3 Results achieved by the PF that have contributed to transformational change, that are institutional and societal level changes that address gender discrimination and exclusion and systematic disadvantage based on other social identities (e.g. through more people centred approaches to planning, efforts to increase voice and accountability to women, poor people and
SMEs etc.).
assessment will be reviewed annually to understand change.
Programme evaluations that capture results related to gendered impacts and benefits from PF programmes or how gender and inclusion has
been considered in programme processes.
Validation with the Programme Managers via Learning Groups structured per family
Mapping findings against the family and Fund ToCs
EQ11: How is the Prosperity Fund learning and why is action on this learning happening more and less successfully?
10.1 What kinds of learning processes are evident within and across programmes and in the Fund? Specifically, where does engagement with learning feature within programme management cycles and fund governance?
10.2 In what instances has learning been
applied?
10.3 Where (if at all) has learning influenced
causal pathways / ToC reviews?
10.4 What enables and inhibits stakeholders to act successfully on learning?
Thematic Evaluation for example on learning, drawing on KIIs and/or survey with the PFMO and programme managers, meeting minutes from ToC workshops and facilitated learning events
Fund Level evaluation that relies on a Year 1 baseline of processes and extent learning has happened. Meta- analysis annually of existing evidence to review progress.
Programme evaluations that generate evidence on how evaluations are being used to inform and improve performance
Units of 'evidence' resulting from evaluation activities (e.g.T0C workshops, KIIs, facilitated learning events) designed as part of the various PF evaluations during implementation.
Description of how programmes have learned as a result of evaluation activities as described in programme / project and fund documentation or through discussions with KI
E&L only (Technical Lead, Portfolio Lead and Thematic Lead)
EQ12: Which Prosperity Fund lessons in translating outputs into intermediate outcomes are sufficiently robust for wider learning?
11.1 What lessons have been generated from the PF?
11.2 Which of these PF lessons tell stakeholders something about relative success in translating outputs into
Thematic Evaluation for example on how learning is being translated more widely drawing on KIIs and/or survey with the PFMO and programme managers, meeting minutes from ToC workshops and facilitated learning events
Fund, programme and project documentation
Data on lessons, especially from analysis related to EQs 1 to 5, as
E&L only (Technical Lead, Portfolio Lead and Thematic Lead)
Pro
sp
erity
Fu
nd
– E
va
lua
tion
an
d L
ea
rnin
g 3
2
Evaluation Questions Indicative Sub Evaluation Questions Evaluative Activities Information Sources Responsibility
intermediate outcomes at programme or project level?
11.3 Which of these programme / project lessons can be grouped (e.g. through triangulation of evidence within or across PF programmes) to show external validity for wider learning?
Fund level evaluation that annually collects and synthesises emerging findings at the Fund level for wider learning and demonstration that they are sufficiently robust
Programme evaluations that generate useful learning that could be sufficiently robust and generalisable to share up to fund level synthesis
reported in E&L evaluation reports
Encourage and help programme and project leads to systematically record and collect their lessons
Where / if different from the above, data on lessons identified through learning-focussed evaluation activities (e.g. workshops, facilitated learning events)
Pro
sp
erity
Fu
nd
– E
va
lua
tion
an
d L
ea
rnin
g 3
3
Annex 2: Evaluation Methodologies
Table 1: Possible methodologies mapped against evaluation questions
Question
Rele
va
nce
Effectiveness
Effic
iency
Susta
ina
bili
ty
Impact
Type o
f q
uestion
Methodologies Evaluation types
RC
T
Diffe
rence
-in-d
iffe
rence
Contr
ibution
an
aly
sis
Realis
t eva
lua
tio
n
Pro
cess tra
cin
g
QC
A
QC
A
VF
M a
na
lysis
Org
anis
atio
nal re
vie
w
Pro
gra
mm
e e
valu
ations
Port
folio
eva
lua
tio
ns
Them
atic e
va
lua
tio
ns
Fund
-level evalu
ation
I. What has been or is likely to be achieved as a result of the PF?
1. What evidence is there that the Prosperity Fund is likely to contribute to the intended outputs and intermediate outcomes in the ToC, as well as any unintended or unexpected effects?
x x x What x x x x x x x X
2. Which types of interventions, sectors and country settings have been more and less successful in contributing to the achievement of primary benefits?
x x x What x x x x x x x x
3. Which types of interventions, sectors and country settings have been more and less successful in contributing to the achievement of secondary benefits?
x x x What x x x x x x x X
4. What evidence is there that Prosperity Fund interventions will be sustainable and ensure environmental sustainability, will be self-financing and lead to inclusive growth that reduces inequality?
x x What x x x x x x x
Pro
sp
erity
Fu
nd
– E
va
lua
tion
an
d L
ea
rnin
g 3
4
Question
Rele
va
nce
Effectiveness
Effic
iency
Susta
ina
bili
ty
Impact
Type o
f q
uestion
Methodologies Evaluation types
RC
T
Diffe
rence
-in-d
iffe
rence
Contr
ibution
an
aly
sis
Realis
t eva
lua
tio
n
Pro
cess tra
cin
g
QC
A
QC
A
VF
M a
na
lysis
Org
anis
atio
nal re
vie
w
Pro
gra
mm
e e
valu
ations
Port
folio
eva
lua
tio
ns
Them
atic e
va
lua
tio
ns
Fund
-level evalu
ation
II. What factors have contributed to or hindered these achievements?
5. What factors have contributed to the achievement of primary benefits and secondary benefits?
x x x How and
why x x x x x x x x
6. How has the balance and relationship between primary and secondary benefits across the portfolio influenced the
achievement of results? x x
How and why
x x x x x X
7. Which assumptions and the causal pathways outlined in the ToC remain valid, which have been adapted and what refinements need to be made?
x x How and
why x x x x x X
8. To what extent is the institutional governance set-up of the Prosperity Fund more or less effective in achieving i)
primary benefits; ii) secondary benefits; iii) other results? x x x x
How and why
x X x x x
9. What types of approaches, governance and management arrangements have been more and less effective for achieving results and demonstrate good approaches to
supporting inclusive growth and VfM?
x x x How and
why x x x X
Pro
sp
erity
Fu
nd
– E
va
lua
tion
an
d L
ea
rnin
g 3
5
Question
Rele
va
nce
Effectiveness
Effic
iency
Susta
ina
bili
ty
Impact
Type o
f q
uestion
Methodologies Evaluation types
RC
T
Diffe
rence
-in-d
iffe
rence
Contr
ibution
an
aly
sis
Realis
t eva
lua
tio
n
Pro
cess tra
cin
g
QC
A
QC
A
VF
M a
na
lysis
Org
anis
atio
nal re
vie
w
Pro
gra
mm
e e
valu
ations
Port
folio
eva
lua
tio
ns
Them
atic e
va
lua
tio
ns
Fund
-level evalu
ation
10. To what extent have the Prosperity Fund interventions contributed to results that support gender equality, women's economic empowerment and social inclusion in line with the UK’s Gender Equality Act and the Prosperity Fund Policy and Guidance and the Prosperity Fund Gender and Inclusion Framework?
x x x How and
why x x x X x x
III. What can be learned from the PF experience to date to improve ongoing and future programming?
11. How is the Prosperity Fund learning and why is action on this learning happening more and less successfully?
x x Learning x x x X
12. Which Prosperity Fund lessons in translating outputs into intermediate outcomes are sufficiently robust for wider learning?
x Learning x x x x
Pro
sp
erity
Fu
nd
– E
va
lua
tion
an
d L
ea
rnin
g 3
6
Table 2: Pros and Cons of Evaluation Methodologies
Design/ approach
Causal Inference Methodology Pros in relation to the PF Cons in relation to the PF
Experimental/ Statistical
Correlation/ counterfactual logic – Experimental randomised control
Randomised control trial (RCT)
• Uses randomised access to programmes as a treatment group and those who do not as a control. It compares the outcomes between the two groups to understand the impact of the programme
• Considered the gold standard in impact evaluation as it can make causal inferences and has the strongest empirical evidence of a treatment’s efficacy
• Randomisation minimises bias and confounding factors
• Logistically challenging in the context of PF as it requires large sample sizes of comparable units of analysis and significant resources.
• Requires a counterfactual to measure change in a population not impacted by the programme
• Less feasible given the scale of impact and the complex relationship between outcomes and programme outputs and timeframe change is anticipated
• Needs to be built into programme design- i.e. programmes need to be designed to randomly deliver services, which can have ethical implications to programme design.
Correlation/ counterfactual logic - Quasi- experimental – counterfactual retrospectively selected control group
Difference and difference, propensity matching,
pipeline approach
• Lacks randomised assignment to a control group, making it a more realistic option for evaluation of complex programmes or relies on statistical techniques to construct an artificial comparison group
• Reliability and validity of evaluation findings.
• The unit of analysis of the programme/context is varied, so context is important; there are too many variables to account for
• Level evidence available may be challenging to assess impact of PF given other factors
• Failure to address questions of ‘how’ and ‘why’ about PF effectiveness.
Theory based design
Generative logic - Identification of causal processes (‘chains’) or supporting factors/ mechanisms in context.
Process Tracing • Suited to investigating and understanding
linkages and causal/contributory pathways – what works, in what circumstances, for whom and why?
• If an aspect of a programme is ineffective, can identify if this is due to theory-failure or implementation-failure.
• Generates learning which can be incorporated and tested as programmes develop.
• Can lack the rigour associated with experimental designs due to potential subjectivity in design, sampling, analysis and interpretation of evidence and the inability to control for confounding factors. A robust approach to sampling and triangulation of the evidence base can mitigate this.
Contribution Analysis • Advantages are the same as for process
tracing, but instead of a focus on robust/thorough hypothesis testing to build a narrative around how change has happened,
• Same disadvantages as for process tracing, but CA also does not discuss explicitly the types and strength of evidence used to come to
Pro
sp
erity
Fu
nd
– E
va
lua
tion
an
d L
ea
rnin
g 3
7
Design/ approach
Causal Inference Methodology Pros in relation to the PF Cons in relation to the PF
CA focuses on building a story of how the programme has contributed to change and then iteratively testing/refining the story.
conclusions about programme contribution to observed outcomes.
Realist evaluation • Considers changes as a result of an
intervention and considers how context influences an outcome
• Explores “what works, for whom, under what circumstances” which can be useful to understand how to adapt interventions
• Can result in generalisability of findings between different programmes given the focus on mechanisms, cross-cutting issues and context
• Places an importance on stakeholders’ involvement to programme development, learning and delivery
• Challenging and requires theoretical understanding to fully explore and articulate programme theory.
• May fail to be comprehensive, particularly as it focuses on clearly defined mechanisms rather than whole and disparate programmes.
• Requires significant amount of data about contexts and outcomes to understand why programmes are expected to work (the processes) differently for different sub-groups or context identified in programme theory.
Organisational Review/Systems review
• Provides information on PF performance and what can be done to improve performance
• Can help understand where there are results/or not results, how implementation quality has influenced delivery
• May be limited in ability to make generalisable comments about the PF portfolio as a whole
• Alone may not generate sufficient information on the change generated by the PF
Case Based Designs
Comparison across and within cases of combinations of causal factors
QCA • QCA could provide rigorous identification and
interpretation of common causal patterns across different PF programmes.
• It would be extremely difficult to apply QCA given the small number of programmes, the number of variables to be explored and the difficulty of defining a successful / unsuccessful outcome in binary terms that is applicable to all the programmes and contexts.
Participatory Designs
Participants/ stakeholders (as co-creators of knowledge and information) provide validation
of causal links.
Collaborative/ Quasi-collaborative evaluation
• Improved programme performance by empowering participants, building capacity and contributing to organisational learning.
• Ensure credibility by enhancing the use and relevance of evaluation results.
• Support inclusion by bringing different perspectives and voices to the process.
• Increased burden for participants
• Resource intensive to design and implement.
• Dependant on evaluation skills of participants (training often required).
• Potential for positive bias in analysis and interpretation of findings.
• Subjectivity and lack of independence.
Pro
sp
erity
Fu
nd
– E
va
lua
tion
an
d L
ea
rnin
g 3
8
Design/ approach
Causal Inference Methodology Pros in relation to the PF Cons in relation to the PF
N/A N/A VfM • Can be useful to understand performance and
improve planning and implementation and support decision making
• Can consider a large number of potential outcomes and can contribute to consensus building and participatory decision-making
• Often relies on benchmarks to understand performance, which may be challenging to gather and contextual differences may make comparison difficult.
• Requires well defined and common benefits to be generated across the programmes.
• Can lead to a focus on evaluation outcomes that are easy to measure and compare, rather than more risky or challenging outcomes, such as complex systems level change as envisioned by the PF.