Developing a Strong DPF Results Framework

February 2020

Outline

I. Introducing the “POLICY and RESULTS MATRIX”

II. Formulating the Program Development Objectives (PDO)

III. Designing Prior Actions

IV.Identifying Results Indicators

V. Results Chain

2

I. POLICY and RESULTS MATRIX

3

Key elements of a DPF

Program Development

Objective

Prior Actions

Results

The Program Dev. Objective is anchored in the country’s own development program/strategy

World Bank supports key policy/ institutional actions, that supports the country in achieving its development objectives

Results of the prior actions are used to monitor and evaluate impact.

4

Policy Matrix is where you start (PD Annex 1)

Prior Actions and Triggers Results

Prior Actions under DPO1 Triggers for DPO2

Pillar A—Program Development Objective A

Prior action #1 (Indicative) Trigger #1 Result Indicator A1Baseline (year):__, Target (year):__

Prior action #2 (Indicative) Trigger #2

Prior action #3

Pillar B—Program Development Objective B

Prior action #4 (Indicative) Trigger #3 Result Indicator B1Baseline (year):__, Target (year):__

Prior action #5 (Indicative) Trigger #4 Result Indicator B2Baseline (year):__, Target (year):__

(Indicative) Trigger #5

Pillar C– Program Development Objective C

Prior action #6 (Indicative) Trigger #6 Result Indicator A1Baseline (year):__, Target (year):__

Prior action #7 (Indicative) Trigger #7

Prior action #8 (Indicative) Trigger #8 Result Indicator A1Baseline (year):__, Target (year):__

Prior action #9

5

II. Formulating the PDO

6

Program Development

Objective

Program Development Objectives

7

Start with the problem / development challenge the government is facing and what it needs to address

PDO:• Anchored in the longer term development

goals of government program • Often composed of pillar objectives• Balance ambition and expected impact that

can be attributed to Bank support

Examples of PDOs

8

PDO Assessment

1. The proposed DPO supports key areas of the reform program with the objectives of: (i) strengthening governance and anti-corruption efforts and putting in place enforcement and evaluation and monitoring mechanisms, implementing key priority areas in the judicial sector to promote the rule of law, and improving public sector accountability and efficiency in the management and use of public resources, and (ii) sharpening competitiveness and enhancing the attractiveness for private investment through improving transparency and governance in the energy sector, reforms in the financial sector, and improving the environment for development of private business.

Lengthy and complex

2. Program Development Objective is to promote sustainable growth through more accountable use of public resources and an improved environment for doing business.

Simple but the text needs to make the links clear

3. The DPO aims to promote sustained growth and ensure inclusivity for the citizens of country XX.

Too general

4. The DPO supports the Government's program of reforms to address the country’s jobs challenges by: (i) modernizing the trade and investment environment; (ii) strengthening systems that protect workers and build resilience; and (iii) improving policies and programs that enhance access to jobs for vulnerable populations.

Clear and focused, text needs to clarify the links to the PAs and results

III. PRIOR ACTIONS

9

Prior Actions

What are Prior Actions?

• An action changes something: it changes incentives, or prices, or governmental structures, etc.

• Policy actions frequently take the form of new laws or regulations or decision rules (e.g. a new tariff-setting mechanism)

• Institutional actions – focus on function, rather than organizational form

• Statements of policy, reports and studies are not usually policy actions. They might list intended future actions, but do not themselves change anything

• The complexity of the program content (prior actions and triggers) should be calibrated to the government’s capacity and “ownership”, and circumstances

• Average of 8 - 9 prior actions per DPO – be selective

10

What are Triggers (in programmatic series)?

• Triggers are indicative prior actions for future operations in the series

• They may be stated with less precision. When the program document is prepared for the next operation, they are made more specific

• Objective is adaptability – to evolving circumstances

• Material changes should be recorded

• Triggers can be dropped, but an explanation needs to be provided without jeopardizing the strength of the matrix especially towards achieving the results and PDO.

• But significant ‘backsliding’ is often penalized at evaluation time. Dropping triggers without trace leaves fewer results; and may be a sign of diminishing government commitment.

11

Plans as policy actions?

12

Can plans, strategies, roadmaps, statements of policy, policy frameworks, etc. be used as prior actions or triggers?

• Not encouraged since typically they are a statement of intent

• Maybe used in programmatic series of DPF, if followed up with implementation leading to results (but not too many)

• More difficult to justify in stand-alone DPF and the last operation of a programmatic series, because:

➢ Plans on their own have few or no results➢ May indicate a lack of understanding of the problem➢ May signal insufficient preparation to have an impact➢ May indicate a lack of commitment

Budget allocations as policy actions?

13

• Not preferred, particularly in stand-alone DPF and the last DPO in a programmatic series

➢ It may indicate lack of commitment

➢ Is not a policy or an institutional action. It is an input to the delivery of a public good in a defined period of time.

➢ It is a temporary / ad hoc measure by nature.

➢ Contrary to the principle of the unity of the budget –because ‘earmarking’

• Maybe, in a programmatic series, if it's part of an approach that will catalyze a sustainable change—but highlight this in the PD.

Examples of Prior Actions

14

Prior Action Assessment

1. Enactment of a Leasing, Factoring and Financing Companies Law, which aims to contribute to the growth of the NBFIs sector and improve access to finance, especially for SMEs.

Substantive and clear

2. Reform the institutional framework governing Telecommunications Companies

Unclear

3. The Government has appointed two independent Board members for the National Electricity Company through a circular

Transactional

4. To strengthen National Electricity Company's governance, a government decree has established the required profile and selection criteria for the independent members of the NEC's Board

Clear action

5. Increase the effectiveness of teaching in secondary schools Unclear

IV. RESULTS

15

Results

Results: What they are and their role in DPFs

• Results tell us how the reforms supported through the DPF will impact the country’s development at the end of the program

• Measured using indicators (with baseline and target)

• Can take the form of outputs or outcomes✓ Outputs are the first level results of reform actions while

outcomes are the second level results.

✓ Outcomes usually take more time to be manifested but are closer to the project development objective

• Preference is for outcomes – more likely to demonstrate that PDO has been reached. However, sometimes difficult to specify or measure outcomes, & assign attribution

16

Results: Links with program development objective

• Key questions: ✓What does the government want to accomplish? Answer is

in PDO ✓How will we know it is achieved? Answer is in results

• Each development objective must be measured by one or more results in the results framework

• Failing this, it is impossible for the task team to evaluate whether the objectives have been attained

• The strength of the link between results and the PDO will be critical when evaluating the program

17

Results: Key features of good results indicators

• Should be focused on measuring the impact at the end of the program

• Specific, not vague

• Time-bound – with baseline and target values, and corresponding dates

• Caused by the prior actions and triggers, i.e. not so distant as to raise questions about attribution

• Different from the prior action that caused it. It should not be a rewording of the prior action

• Quantitative, not qualitative, whenever possible

• Target value should be realistic

18

Results: Examples of results indicators

19

Examples of Prior Actions Results indicator Assessment

Approval of Water Act supports water resources management including DRM and climate change adaptation

Water Harvesting andStorage Authority established and operationalized forimproved water resource and drought management

Weak / vague result indicator

Adoption of (i) updated operation standards for sanitary landfills; and (ii) updated regulations for complementary solid waste treatment activities

Municipalities disposing solid waste insanitary landfills in compliance with new legal framework [Baseline 2015: 874; Target 2018: 916]

Reasonable, measurable (?) indicator.

Cabinet has approved the introduction of a tariff for residential water use

Number of households with access to reticulated water 24/7 Baseline: 0 (2017)Target: 400 (2021)

Demonstrate attribution. Result could be share of population? Also for percentage of women reporting a positive impact ?

Borrower has adopted an Action Plan on Drinking Water Sector Reforms, including the creation of a new nation-wide lease contract

Number of Water User Associations setting fees according to business plansBaseline: 0Target: > 5

Two actions (one an intent). Reasonable result indicator: specific, timebound, seems realistic, linked to the prior action, quantitative but also more output than outcome focused

PDO

PAs

Results

20

V. Results Chain

Results chain: Key elements

• Results chain illustrates the causal sequence of the reform program being supported, from prior actions to results to PDO

• In designing DPFs, it is important to assess the strength of a results chain

• A strong results chain provides greater assurance that the reform program will achieve the intended development objectives

• Some of the key elements of a strong results chain are as follows:

✓There must be a logical and strong link amongst its elements

✓Timing of prior actions and triggers must be appropriate for the achievement of the intended result

✓The result(s) must be sufficient to measure achievement of the PDO

21

Resources (I) – OPCS DPF Guidance

22

Type “DPF/” in your browser

➢ Policies ➢ Processing instructions➢ Templates ➢ Guidance notes➢ Examples➢ and more!

Resources (II) –Learning Programs

23

•DPF E-Learning

•DPF Academy (2-day workshop)

• ICR for DPF Operations (90-min. Clinic coming soon)

•Designing a Better DPF Results Framework

•Recordings of previous DPF Clinics

https://wbg.sabacloud.com/Saba/Web_spf/NA1PRD0002/common/ledetail/20133659/latestversionhttps://wbg.sabacloud.com/Saba/Web_spf/NA1PRD0002/common/ledetail/20204698https://wbg.sabacloud.com/Saba/Web_spf/NA1PRD0002/common/ledetail/00034946/latestversionhttps://wbg.sabacloud.com/Saba/Web_spf/NA1PRD0002/common/ledetail/cours000000000018322https://olc.worldbank.org/content/improve-your-understanding-bank-policies-guidance-and-procedures-they-relate-development

RESERVE SLIDES

24

Tips on designing PAs

DOs

Use only the highest-priority actions as prior actions and triggers. Limit the set of policy and institutional measures to those that are essential and practicable

given institutional and political constraints

Ensure that each prior action contributes to at least one result

Ensure the PA/trigger thread builds on each other

DON’Ts

Don’t use outputs, outcomes, transactions or ad hoc measures as

prior actions

Don’t define prior actions or triggers that are vague or difficult to monitor

Don’t establish a prior action or trigger that is not grounded in prior analytical

work

25

Do’s and Don’ts of designing Results

DO

Check that the operation’s results are consistent and

linked with those in the CPF.

Use quantitative indicators, including baselines and

targets, whenever possible.

Include at least one end-of-program result for each

policy area of the program

DON’T

Don’t try to identify a separate end-of-program result for each action

Don’t use policy and institutional actions as results

Don’t include results that are not directly influenced by actions that are part of the

program

Don’t include results that are beyond the timeframe of the program.

26

Results: Systems for monitoring and evaluation

• Borrowers are owners of Bank-financed reform programs, and hence are responsible for ensuring that they achieve their development objectives

• M&E is therefore the responsibility of the client Government

• The Bank has an invested interest:

✓ As a development agency, we want to see our clients achieve their development objectives

✓ We are interested in evaluating our own contribution to results

• The program document assesses the client’s M&E systems

• Due to capacity deficiencies, the Bank team works closely with Government counterparts in M&E

27

Results chain in the main text

• Once the prior actions, triggers and results have been clarified, the write-up of the remainder of the document is easier.

• From the matrix to the main text:

✓ Rationale for the prior actions in the main text. The main text should explain briefly the rationale for the prior actions and triggers. There is no need for lengthy descriptions of the sector, or its history, or the fine details about the discussions with the Bank, etc.

✓ Results in the main text. The main text should mention briefly the results expected from the prior actions and triggers.

✓ Linkages in the main text. If the linkages between the prior actions and the triggers are not obvious, the main text should make them explicit.

28

Designing a programmatic series

The government has submitted to the Parliament draft Public Debt Management Law

Government has issued implementing regulations including on the establishment of the new Debt Management Unit within the MoF

The government approves the first Medium-term debt management strategy

The debt unit publishes the annual debt management report including a table on contingent liabilities

29

DPO1 DPO2 DPO3

Indonesia: using DPF to address critical constraints

Reforming the Indonesian maritime logistics sector ($400 m, FY17, P158140). To reduce costs of logistics, by privatization of ports and eliminating importer licenses.

Sustainable and Inclusive Energy DPF ($500 m, FY16, P154291). To support reforms that: (a) reduce the fiscal cost of electricity; and (b) expand access to modern energy.

Indonesia Fiscal Reform DPL 2 ($300 m, FY18, P161475). To support fiscal reforms that: broaden the tax base, improve revenues, and make public spending more efficient.

30

http://projects.worldbank.org/P158140?lang=enhttp://www.projects.worldbank.org/P154291?lang=enhttp://projects.worldbank.org/P161475?lang=en

Results: Examples of results indicators

31

Prior action (2019) Results indicator Assessment

Minister of Finance has signed regional power trading agreement and agreement is effective

1. Regional power trading agreement signedBaseline: No (2019)Target: Yes (2020)

Result is simply a rewording of the prior action – what is the change?

Minister of Finance has signed regional power trading agreement and agreement is effective

2. Significant improvement in power supply

Result is vague & not specific; it is not time bound with no baseline and target values

Cabinet has approved privatization of power utility company - Transaction

3. Increase in share of rural households with access to electricityBaseline: 10% (2019)Target: 100% (2020)

Difficult to attribute result to prior action, unrealistic target

Minister of Finance has signed regional power trading agreement and agreement is effective

4. Reduction in hours of power supply load sheddingBaseline: 10 hours a month (2019)Target: 5 hours a month (2020)

Good result indicator: specific, timebound, seems realistic, linked to the prior action, quantitative