Promoting Basic Services Program (Phase-3) in Ethiopia...

Promoting Basic Services Phase III(PBS3) Project

in Ethiopia - Process Evaluation

Final ReportMarch, 2016

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Evaluation TeamDr. Swapan Kanti Chaudhuri (International Expert & Team Leader)

Raya Abagodu (National Expert)

Tilahun Wolde (National Research Associate)

Sanjay Tripathi (International Research Associate)

Ridhi Kapur (International Research Associate)

Sunil Verma (Program Manager)

IPE Global Limited B-84, Defence Colony, New Delhi, IndiaPin-110024www.ipeglobal.comPhone (Ethiopia Office): +251 92868 0596/9132680

Promoting Basic Services Phase III(PBS3) Project

in Ethiopia - Process Evaluation

Final Report, March 2016

Acknowledgement

The study team would like to express a deep sense of gratitude to the PBS Secretariat (coordinator and other experts), the World Bank (task team lead and staff, including sector advisors), DfID (task team lead and staff, including sector advisors), European Union (donors co-chair), African Development Bank, Italian Development Cooperation, and Austrian Development Cooperation for their guidance and valuable contributions in the preparation of this report. Without their support it would not have been possible to complete this report. The team members would also like to thank the head and staff of the Management Agency for providing all the details about implementation of the Ethiopia Social Accountability Program. The team is grateful to DfID and the PBS Secretariat which provided the funding for the study.

The study team is extremely grateful to the concerned officials of the Ministry of Finance and Economic Development, Ministry of Education, Ministry of Health, Ministry of Water, Irrigation and Electricity, and the Central Statistical Agency for extending support, and providing valuable insights into the program.

A good number of government officials at the region and woreda level have generously contributed towards completion of this report. The study team is indebted to all of them. Last but not the least, the team members would also like to thank all the community members who participated and enriched the focus group discussions conducted at kebele level.

Disclaimer: Opinions and views expressed in this document are entirely those of the authors, and do not reflect opinions of the Government of Ethiopia, Development Partners including The World Bank, or any other civil society organizations.

Currency Equivalent

Currency Unit: Ethiopian Birr

1 USD = 21.09 ETB as on 29 November 2015

Fiscal Year: July 8 to July 7

Ethiopian Fiscal Year

European Fiscal Year

1999 2006/07 2000 2007/08 2001 2008/09 2002 2009/10 2003 2010/11 2004 2011/12 2005 2012/13 2006 2013/14 2007 2014/15 2008 2015/16

Abbreviations and Acronyms

AfDB African Development Bank AGP Agricultural Growth Project ANC Antenatal Care BoFED Bureau of Finance & Economic Development CDP Community Development Program COPCO Channel One Programs Coordination Office CSA Central Statistical Agency CSO Civil Society Organization DA Development Agent DAC Development Assistance Committee DAG Development Assistance Group DfID Department of International Development (UK) DP Development Partner DQA Data Quality Assessment EDHS Ethiopia Demographic and Health Survey EDQAF Ethiopia Data Quality Assessment Framework EFY Ethiopian Financial Year EIO Ethiopian Institute of Ombudsman EMIS Education Management Information System EQ Evaluation Question ERA Ethiopian Roads Authority ESAP Ethiopia Social Accountability Program ESDP Education Sector Development Program ESPES Enhancing Shared Prosperity through Equitable Services ETB Ethiopian Birr EU European Union FBG Federal Block Grant FEACC Federal Ethics and Anti-Corruption Commission FGD Focus Group Discussion FPPPAA Federal Public Procurement and Property Administration Agency FTA Financial Transparency and Accountability GBS General Budget Support GDP Gross Domestic Product GEQIP General Education Quality Improvement Program GER Gross Enrolment Ratio GoE Government of Ethiopia GPI Gender Parity Index GRM Grievance Redress Mechanism

GTP Growth and Transformation Plan HEW Health Extension Worker HMIS Health Management Information System HSDP Health Sector Development Program IBEX Integrated Budget and Expenditure System IGFT Inter Governmental Fiscal Transfer IMF International Monetary Fund IMR Infant Mortality Rate JBAR Joint Budget and Aid Review JRIS Joint Review and Implementation Support KII Key Informant Interview LIG Local Investment Grant M&E Monitoring & Evaluation MA Management Agent MDGs Millennium Development Goals MEFF Macroeconomic and Fiscal Framework MforR Managing for Results MIS Management Information System MMR Maternal Mortality Ratio MoA Ministry of Agriculture MoE Ministry of Education MoFED Ministry of Finance & Economic Development MoH Ministry of Health MoU Memorandum of Understanding MTR Mid-Term Review NER Net Enrolment Ratio OECD The Organization for Economic Co-operation and Development OFAG Office of the Federal Auditor General ORAG Office of the Regional Auditor General PAD Project Appraisal Document PBS Promoting Basic Services (formerly Protecting Basic Services) PDO Project Development Objectives PEFA Public Expenditure and Financial Accountability PFM Public Financial Management PforR Program for Results PSCAP Public Sector Capacity Building Program PSIA Poverty and Social Impact Assessment PSNP Productive Safety Net Project PTR Pupil Teacher Ratio SA Social Accountability SAC Social Accountability Committee

SAIP Social Accountability Implementing Partners SDG Sustainable Development Goal SDI Service Delivery Indicator SDS Service Delivery Secretariat SNNPR South Nations, Nationalities and People’s Region ToR Terms of Reference TWG Technical Working Group U5MR Under Five Mortality Rate UNICEF United Nations Children's Fund URRAP Universal Rural Road Access Program USAID United States Agency for International Development USD United States Dollar VfM Value-for-Money WASH Water Sanitation & Hygiene WoFED Woreda office of Finance & Economic Development

Contents

Executive Summary ................................................................................................................................ 1Chapter 1: Introduction ........................................................................................................................ 1

1.1 Background ................................................................................................................................ 11.2 Objectives of the Assignment .................................................................................................... 21.3 The Present Report .................................................................................................................... 2

Chapter 2: An Overview of the PBS-3 Program ................................................................................. 32.1 Key Features of the PBS-3 Program (Dec 2012 – Jan 2019) .................................................... 3

2.1.1 Development objectives ....................................................................................................... 32.1.2 Program components ............................................................................................................ 32.1.3 Core principles, tests and reviews ........................................................................................ 42.1.4 Program implementation arrangement ................................................................................. 6

2.2 Realignment of PBS-3 ............................................................................................................... 72.2.1 New program ESPES ........................................................................................................... 72.2.2 Other restructuring ................................................................................................................ 7

2.3 Summary .................................................................................................................................... 7Chapter 3: Approach and Methodology .............................................................................................. 9

3.1 Evaluation Framework ............................................................................................................... 93.2 Methodology .............................................................................................................................. 9

3.2.1 Desk research ........................................................................................................................ 93.2.2 Federal level interviews ...................................................................................................... 103.2.3 Field mission in select regions and woredas ...................................................................... 103.2.4 Data analysis ....................................................................................................................... 11

3.3 Limitations of the Study .......................................................................................................... 113.4 Summary .................................................................................................................................. 11

Chapter 4: Relevance and Coherence of the PBS-3 Program ............................................................ 124.1 Country Context at Appraisal .................................................................................................. 12

4.1.1 Political development ......................................................................................................... 124.1.2 Economic development ...................................................................................................... 134.1.3 Progress towards MDGs ..................................................................................................... 134.1.4 Recurrent challenges of basic service delivery ................................................................... 13

4.2 Program Relevance .................................................................................................................. 134.2.1 Validity of program development objective ....................................................................... 134.2.2 Relevance of PBS-3 activities and outputs in achieving PDO ........................................... 144.2.3 Relevance of focusing on five sectors ................................................................................ 144.2.4 Discontinuation with LIG and health components ............................................................. 144.2.5 Relevance of core principles .............................................................................................. 16

4.2.6 Appropriateness of results framework ................................................................................ 164.3 Program Coherence ................................................................................................................. 174.4 Summary of Findings .............................................................................................................. 18

Chapter 5: Program Effectiveness (Progress in Achieving Results) ................................................. 195.1 Effectiveness of Basic Service Delivery .................................................................................. 19

5.1.1 Progress towards achieving higher-level and PDO targets ................................................ 195.1.2 Sector wise progress performance ...................................................................................... 20

5.2 Mission Findings on Basic Service Delivery ........................................................................... 225.2.1 Key findings based on staff interviews at select regions/woredas ..................................... 225.2.2 Quality of service delivery (FGD findings) ........................................................................ 25

5.3 Progress in Strengthening Local Accountability and Transparency ........................................ 265.4.1 Citizen’s engagement ......................................................................................................... 265.4.2 Public financial management and procurement ................................................................. 295.4.3 Managing for results ........................................................................................................... 30

5.4 Summary of Findings .............................................................................................................. 31Chapter 6: Aid Effectiveness of the PBS Program ............................................................................ 33

6.1 Aid Landscape in Ethiopia ....................................................................................................... 336.2 Aid Effectiveness of the PBS program .................................................................................... 346.3 Donors’ Engagement in Policy Dialogue with the Government ............................................. 356.4 Recent Initiatives for Donors’ Harmonization ........................................................................ 366.5 Summary of Findings .............................................................................................................. 37

Chapter 7: Program Efficiency .......................................................................................................... 387.1 Assessment of Value-for-Money ............................................................................................. 387.2 Efficacy of the Program Implementation ................................................................................ 407.3 Risk Management .................................................................................................................... 427.4 Summary of Findings .............................................................................................................. 43

Chapter 8: Impact and Sustainability ................................................................................................. 448.1 Impacts of the PBS Program .................................................................................................... 44

8.1.1 Impact of block grant spending on sector outcomes (PSIA study) .................................... 448.1.2 Progress towards achieving MDG targets .......................................................................... 448.1.3 Unintended or negative impact of the PBS-3 program ...................................................... 45

8.2 Sustainability of Basic Services ............................................................................................... 458.2.1 Fiscal sustainability ............................................................................................................ 458.2.2 Institutional sustainability .................................................................................................. 46

8.3 Summary .................................................................................................................................. 47Chapter 9: Lessons Learned and Recommendations ......................................................................... 48

9.1 Lessons Learned ...................................................................................................................... 48

9.2 Recommendations .................................................................................................................... 499.2.1 Focusing on education, health, and agriculture sectors where program impacts are more 499.2.2 Setting up a high-level steering committee for strategic discussion and decision making 499.2.3 Strengthening donors’ harmonization ................................................................................ 499.2.4 Benchmarking service delivery .......................................................................................... 519.2.5 Conducting beneficiary survey ........................................................................................... 529.2.6 Regions to check five per cent administrative data related to basic services ..................... 529.2.7 Expediting the study on service cost estimation and resource mobilisation ...................... 529.2.8 Institutionalization of social accountability activities ........................................................ 52

Select Bibliography ............................................................................................................................... 53Annex-1: Terms of Reference ............................................................................................................... 56Annex-2: List of People Met ................................................................................................................. 66Annex-3: Evolution of the PBS Program .............................................................................................. 71Annex-4: Progress Towards Equitable Services in Education and Health ........................................... 73Annex-5: PBS-3 Results Data (Achievements vs. Targets) .................................................................. 76Annex-6: Progress Towards Achieving MDGs .................................................................................... 82

Figures Figure 4.1: Theory of change for protecting/promoting basic services ............................................ 15Figure 7.1: Sector shares in expenditure at woreda level (2011-12 to 2013-14) .............................. 38Figure 7.2: Cost structure of basic service (2011-12 to 2013-14) .................................................... 38Figure 8.1: Sensitivity of projected budget deficit estimates ............................................................ 46

Table

Table 2.1: Core principles, tests, and reviews of PBS-3 ..................................................................... 5Table 3.1: Evaluation framework and key questions .......................................................................... 9Table 3.2: Sample regions and woredas ............................................................................................ 10Table 5.1: Achievement of targets in 2012-13 & 2013-14 ............................................................... 19Table 5.2: Performance of schools and health facilities as per service bureaus ............................... 23Table 5.3: Service improvements due to social accountability facilitations ..................................... 28Table 8.1: Projected expenditure, revenue, and budget deficit ......................................................... 45Table 9.1: Proposed framework of donors’ harmonization .............................................................. 50Table 9.2: Proposed objectives and activities of the newly created SDS ......................................... 51

Executive Summary

This report presents findings of process evaluation of the Promoting Basic Services (Phase-3) Project, which was undertaken during the second half of 2015. The evaluation was conducted in terms of six criteria (relevance, effectiveness, efficiency, impact, sustainability, and coherence) and twelve key questions. Given the nature of evaluation, lesser emphasis was given on impact assessment. The methodology included: desk research; key informant interviews with the concerned ministries, federal agencies, and the development partners; group interviews with key staff of Bureau of Finance and Economic Development and five sector bureaus in four regions (SNNPR, Oromiya, Somali, and Benishangul Gumuz); group interviews with key staff of Woreda Office of Finance and Economic Development and sector bureaus in eight woredas (two in each selected region); and focus group discussions with 188 community members across eight kebeles (one in each selected woreda). A major limitation of the present study is that the field data has been collected from a few regions, woredas, and kebeles. The sample size could not be increased due to time and resource constraints.

The key findings of the process evaluation are summarised in the table given below:

Key findings

Relevance & Coherence

• The project development objective of PBS-3 was valid and appropriate in the country context at appraisal, but the program did not address adequately inequities in basic services across regions and across income/wealth quintiles.

• The theory of change underscores the relevance of activities/outputs of PBS-3 in achieving project development objective. It also highlights strong complementarity between the PBS program and the sector specific programs. Integration of PBS with sector programs should be one of the key considerations for on-going efforts towards donors’ harmonization.

• The water and road sectors require more capital investments, while the PBS funding through block grants supports only revenue expenditure. Thus, the PBS program has hardly benefited these two capital-intensive sectors. The present evaluation is of the opinion that PBS-3 could have covered only education, health, and agriculture sectors instead of thinly spreading resources.

• The core principles of PBS-3 are of great relevance, but there is scope for improving sustainability, equity and predictability reviews.

• The results framework suffers from some shortcomings such as inappropriate higher-level health indicators; lack of technical note to define indicator for measurement; use of double data source without setting comparable targets; and data inconsistencies.

• The Sub-Program A and Sub-Program B of PBS-3 are coherent with the fifth and sixth pillars of the Growth and Transformation Plan respectively. Both the components also complement each other. However, the scope of institutional reforms under Sub-Program B embraces too many challenging areas.

Effectiveness • Of the seven higher-level indicators, 43 per cent targets were achieved in 2012-13 and 57 per cent in 2013-14. Similarly, of the six PDO indicators, 50 per cent and 60 per cent targets were achieved in 2012-13 and 2013-14 respectively. The overall achievements during the period under consideration may be rated moderate. Sector performance

• Net enrolment ratio at Grade 5-8 remained consistently low (less than 50 per cent) and fell far short of the program targets. The proportion of qualified teachers at Grade 1-8, another key indicator, improved over the years but remained significantly below the targets in 2012-13 and 2013-14. The targets related to gender parity index, the student-teacher ratio and number of qualified teachers were fully met in both the years.

• Penta-3 vaccine coverage increased over the years but targets were underachieved in 2012-13 and 2013-14. The ANC coverage overachieved the targets. The population per health extension worker was below the norm (one extension worker per 2,500 population) in 2012-13 but it improved in 2013-14 almost reaching the norm.

• Increase in development agents with required specialisation did not meet the targets in both the years, while the targets for food crops productivity and outreach to direct beneficiaries

Process Evaluation of PBS-3, Final Report: Executive Summary

2

Key findings

fully met the targets in both the years. • Rural water supply coverage and reduction in non-functional water schemes in 2012-13

and 2013-14 remained far below the targets, while all the targets in road sector were achieved during that period. Findings of the field mission

• Most of the woredas grade performance of their primary/secondary schools, while lesser number of woredas evaluates health facilities. The findings indicate opportunity for developing an appropriate results based and incentivised service management system.

• The woredas are faced with some recurrent challenges/constraints, such as shortage of skilled/trained manpower, high staff turnover, and budget constraint to finance operating expenditure and capital investments. Besides, there are some sector specific problems.

• Most of the community members in select kebeles feel that there has been marginal improvement in the quality of basic services compared to earlier years. In their opinion, education, health, agriculture, and sanitation services have improved compared to water and rural road services. Progress in strengthening local accountability and transparency

• The targets of financial transparency and accountability and social accountability were achieved in 2012-13 and 2013-14. Anecdotal evidence also suggests that citizen acts on dissemination of budget and expenditure data. Similarly, social accountability facilitations along with other interventions have brought about improvements in basic service delivery. However, community consultation by woreda officials for improving service delivery has not progressed as expected. In recent period, there has been satisfactory progress in linkage between financial transparency and accountability and social accountability activities.

• GRM offices are present in all regions except in Afar. Five regions (as against a target of six regions) prepared and adopted standardised manual and guidelines in 2012-13. In 2013-14, the number of such regions increased to eight and the target was achieved. Grievance redress mechanisms at regional, zone, and woreda levels are facing several challenges (e.g. lack of independence; weak enforcement of decisions). Besides, citizens are generally not satisfied with redress of complaints.

• The public financial management at woreda level has progressed well. The achievements inter alia include extensive roll out IBEX; improved audit coverage by the Office of Regional Auditor General; and hundred per cent audit coverage by the Office of the Federal Auditor General. However, a large number of woredas do not take follow up actions on audit findings. Procurement is another significant challenge at woreda level.

• Under managing for results component, the updating and sharing of PBS results in every Joint Review and Implementation Support has been satisfactory. But the works related to strengthening multi-sector management information systems and conducting some analytical studies have not progressed well. Due to slow progress, forward work plan of this component has been redrawn and prioritised, and budget allocation has been reduced. Aid Effectiveness

• PBS program meets very well the aid effectiveness principles of the Paris Declaration. • The semi-annual Joint Review and Implementation Support and Joint Budget and Aid

Review do not provide any scope for strategic discussion and decision-making. This calls for rethinking on the existing format of semi-annual joint review forum.

• Given the realignment of donors’ support to the PBS program, the donors are working towards developing a common framework for donors’ harmonization. The work is still in-progress. The PBS secretariat has been reconstituted as Service Delivery Secretariat, which is required to play a critical role in donors’ harmonization.

Efficiency • The PBS program has moderately achieved value-for-money. Unlike the DfID study, the present evaluation does not consider a declining trend in the real salary as ‘economy’ of staff input costs. In the Ethiopian context, low salary structure is one of the causal factors for staff turnover in the public sector. The value-for-money analysis also reveals that ‘efficiency’ and ‘equity’ values have not been achieved at the desired level.

• The Bank’s leadership and implementation support has been satisfactory. GoE has shown sustained ownership and commitment for the program. PBS secretariat has been quite


3

Key findings

effective in achieving donors’ harmonization and coordination. • The bank has developed environment and social screening tool to manage potential risks

associated with any operational interface between the Bank-funded projects and community development program of the regions. The federal government has already approved the screening tool.

Impact • The poverty and social impact assessment study of the World Bank finds positive impacts of woreda-level spending in achieving outcomes in education, health, and agriculture sectors. It also claims that benefits of woreda-level spending on education and health have percolated to all income levels, and per capita expenditure favoured disadvantaged regions and ethnic groups.

• The present evaluation has tracked progress towards millennium development goals 1 to 5 applying counterfactual analysis and using latest available data. The progress has been satisfactory except for maternal health, which remained completely off the track. It may be reasonable to say that the PBS program with its support for expansion of basic services has contributed in this progress.

• There is no specific reference or proven records of any unintended or negative impacts of the PBS-3 program except the alleged link of the program with human right abuses in Gambella. The allegation was not found true, but it has brought the program under intense international scrutiny.

Sustainability • The medium term projections (2015-16 to 2019-20) undertaken in the present evaluation study shows that if the current fiscal conditions continue, provision of basic services for the growing population would require, in addition to GoE’s own funding, donors’ support of USD 150-200 million per year.

• The PBS program has made significant strides towards institutional sustainability in public financial management, citizen engagement through financial and social accountability activities, and establishing grievance redress mechanisms. However, more reforms are still required, especially for institutionalisation of social accountability initiatives.

Lessons learned

• Decentralised service delivery supported through inter governmental transfers is an effective development model to reduce poverty and achieve other millennium development goals.

• The PBS program through its partnership and pooled-fund approach offers an effective alternative to general budget support system.

• Citizen engagement activities promote direct accountability of the service providers, and improve service delivery.

• Use of the country systems has been crucial for gaining the government’s ownership of the program.

• A dedicated resource allocation by the development partners to set up a PBS secretariat is an essential step forward to achieve donors’ coordination and harmonization.

• Pilots should be undertaken with a perspective plan on how the experience and results of pilots will be scaled up to a national level.

• Decentralised basic service delivery is a complex process, and it must be supported by deep-rooted institutional reforms at sub-national level.

• Failure to assess environmental and social risks may put the program at reputational risk.

Recommendations

• Focusing on education, health, and agriculture sectors where program impacts are more. • Setting up a high-level program steering committee for strategic discussion and decision-

making. • Strengthening donors harmonization at two levels:


4

− Sector level (Level-1): Developing a single integrated plan/budget, and common results framework, and conducting a combined joint review mission for the sector

− National level (Level-2): Synchronization of multi-sector programs with the similar or complementary program interventions, and developing a common strategy and work plan for the development partners

− Redefining objectives and activities of the service delivery secretariat − Ensuring required technical skills by hiring full-term or short-term experts

(economist; public financial management specialist; environmental and safeguard specialist; monitoring and evaluation expert; knowledge and communication expert)

• Benchmarking service delivery at woreda and service provider levels • Conducting yearly beneficiary survey in each region using standard tools, and integrating the

regional results to get a national perspective • Regions to check five per cent sector data every year in two to three sample woredas with the

technical support from CSA • Expediting the exercise on estimation of cost of basic services and resource mobilisation • Pending development of a concrete plan for institutionalisation of social accountability

activities, undertaking the following: − Sustained awareness campaign for citizens steered by local administrators and

council members − Sensitizing council members, government officials, and service providers about

citizens’ right to services and their responsibilities, and building their capacity to use of social accountability tools

− Engaging the existing community based organizations (e.g. parent teachers associations, health boards, water committees) in social accountability activities and building their capacity to use social accountability tools

− Harmonizing social accountability program funded by other development partners (e.g. United States Agency for International Development) with the Ethiopia Social Accountability Program

Chapter 1: Introduction

1.1 Background

1. Ethiopia is a geographically and culturally diverse country with an estimated population of 90 million1. The country has followed a development state model, with poverty reduction as a core theme of its growth and transformation plan (GTP). It has achieved remarkable economic growth (averaging 10.6 per cent over 2006-2014), and made substantial progress in achieving millennium development goals (MDGs) including poverty reduction. However, despite growth and development, Ethiopia remains one of the poorest countries in the world. The country’s per capita income (USD 550) is much lower than the Sub-Saharan Africa’s average income (USD 1, 638)2.

2. The Government of Ethiopia (GoE) has a long history of working with the development partners (DPs), which provided substantial funds to the government in the form of general budget support (GBS). Following the post-election crisis in 2005, some DPs decided to suspend budget support, while other including the World Bank (hereinafter referred to as Bank), Department for International Development (DfID), and African Development Bank (AfDB) looked for a new funding mechanism to support the government since complete withdrawal of budget support might cause reversal of the country’s economic and human development progress achieved till date. In 2006, the protection of basic services (PBS) program became the new funding mechanism. Initially, the donors considered the program for preparing ground for resumption of GBS, but as time progressed both GoE and DPs started appreciating contributions of PBS in improving decentralised basic services, and the program continued through phases3.

3. The first phase (PBS-1) ran from June 2006 to December 2009, and it supported the following basic sectors: education, health, agriculture, and water supply and sanitation. The program completion cost was USD 2,958 million against budget estimates of USD 1,929 million4. The second phase (PBS-2) covered the period from May 2009 to January 2013. It expanded the definition of basic services to include rural roads. The program was completed at USD 3,763 million compared to budget estimates of USD 2,707 million5.

4. Effective from December 2012, the third phase (PBS-3) began and it is now scheduled to close on 7 January 2019. The PBS-3 program is rechristened as promoting basic services, reflecting the fact that both the government and DPs have demonstrated their commitments to protect basic service delivery and, hence, the extended phase therefore stands for promoting basic services. The revised budget estimate of the PBS-3 program is USD 1215.74 million, of which USD 924.01 million (76 per cent) has been disbursed as of 19 May 20156.

5. Very recently, the PBS-3 has witnessed several changing circumstances, such as (i) withdrawal of DfID’s funding support with effect from 1 July 2015, causing a financing gap of USD 162.4 million7; (ii) realignment of the Bank’s support to basic services through a new program titled Enhancing Shared Prosperity through Equitable Services (ESPES) at an estimated IDA funding of USD 600 million; (iii) AfDB’s proposed sector budget support program called Basic Services Transformation Program (BSTP) at an estimated cost of UA 180 million (USD 252 million); and (iv) the decision of DfID and the European Union (EU) to focus on direct engagement in basic service sectors. Notwithstanding all these realignments, the PBS donors have recognised the urgent need for

1 The United Nations estimates that the population will reach 130 million by 2025. 2 Source: Atlas GNI, 2014 3 See World Bank (December 2013): Protection of Basic Service Project: Project Performance Assessment Report, Para 2.3, p 6 4 MoFED (21 June 2013): Protection of Basic Services, Phase II – Government’s Implementation Completion Report, Table 1, p 9 5 Ibid 6 World Bank (20 July 2015): Restructuring Paper 7 Ibid. In February 2015, DfID communicated to the Bank its decision to withdraw support from the PBS-3.

Process Evaluation of PBS-3, Final Report: Introduction

2

donors’ harmonisation and sustained dialogue on decentralised service delivery8. At present, DPs (the Bank, DfID, AfDB, EU, Austria Development Agency, and Italian Development Cooperation) are working together to develop a common framework and mechanism for donors’ harmonization.

1.2 Objectives of the Assignment

6. In the above background, the present assignment involves an independent process evaluation of the PBS-3 program. The terms of reference (ToR) of this evaluation is given in Annex-1.

7. The specific objectives of the assignment are the following:

• Undertaking process evaluation of the PBS-3 program across six dimensions viz. relevance, effectiveness, efficiency, impact, sustainability, and coherence; and

• Providing recommendations to improve the on-going program implementation, and to inform future program design and resource allocations.

1.3 The Present Report

8. The process evaluation of the PBS-3 was conducted during the second half of 2015. The findings and recommendations were first presented in the Draft Report (dated 11 December 2015). Based on the comments and suggestions received from the donors, the draft has been finalised and presented in this report, consisting of the following chapters:

Chapter 2: An overview of the PBS-3 program

Chapter 3: Approach and methodology Chapter 4: Relevance and coherence of the PBS-3 program

Chapter 5: Program effectiveness (progress in achieving results) Chapter 6: Aid effectiveness of the PBS program

Chapter 7: Program efficiency Chapter 8: Impact and sustainability

Chapter 9: Lessons learned and recommendations

Annexes

8 Draft workshop report (28 January 2016) on Services Delivery Harmonization

Chapter 2: An Overview of the PBS-3 Program

This chapter presents key features of the PBS-3, and provides an update on the program’s recent realignment. A brief note on evolution of the PBS program is presented in Annex-3.

2.1 Key Features of the PBS-3 Program (Dec 2012 – Jan 2019) 2.1.1 Development objectives

9. The PBS-3 program has evolved leveraging on the core architecture and the implementation experience of the earlier two phases. Its project development objective (PDO) is essentially the same as pursued under PBS-1&2. In specific terms, PBS-3 is designed “to contribute to the higher-level objective of expanding access and improving the quality of basic services by funding block grants that ensure adequate staffing and operations, and by strengthening the capacity, transparency, accountability and financial management of regional governments and authorities at woreda level”.

2.1.2 Program components

10. Originally, the PBS-3 program included three components viz. Sub-Program A (block grants), Sub-Program B (strengthening local accountability and transparency systems), and Sub-Program C (results enhancement fund). The Sub-Programs C was dropped due to implementation challenges. Recently, a new sub-component has been added to the Sub-Program B.

11. Sub-Program A: This is the main funding component of the program. The funds are provided in the form of contribution to the government’s inter-governmental fiscal transfers (IGFT), or block grants, from federal to the regions (excluding Addis Ababa), and from regions to the woredas. At woreda level, local administration has discretion to decide allocation of block grants to the five basic sectors based on its own needs.

12. The block grants provided under the program can be used for financing only the woreda-level recurrent expenditure (salaries and operating expenditure) in the five basic services viz. education, health, agriculture, water supply and sanitation, and rural roads. With the availability of increased block grants, the woreda administration goes for hiring additional frontline service providers, such as primary school teachers, health extension workers (HEW), and development agents (DAs). They also use block grants to pay salaries of government experts who oversee and manage development of new water schemes. Similarly, the woredas pay staff salary to set up and operate road desks that support rural roads construction.

13. Sub-program B: This component supports strengthening local accountability and transparency systems through three interrelated sub-components viz. (i) citizen’s engagement (sub-component B1); (ii) local public financial management and procurement (sub-component B2); and (iii) managing for results (sub-component B3).

14. The citizen’s engagement component seeks to help woredas improve quality of service delivery through an institutionalized system for financial transparency and accountability (FTA), social accountability (SA), and grievance redress mechanism (GRM). While the FTA component aims at deepening the citizens' access to information on budget, expenditure and service delivery, the SA component creates demand for greater transparency by encouraging citizens’ participation in basic service delivery through engagement and support of civil societies, non-government organizations called social accountability implementing partners (SAIPs)9. The Ethiopia Social Accountability

9 The SA process involves: (i) citizen’s access to information about service standards, plan targets, and budgets; (ii) training of citizen on how to use SA tools (community score card; participatory planning and budgeting; citizen report card; gender responsive budgeting; and public expenditure tracking survey) for assessing access and quality of service delivery vis-à-vis standard/plan/budget; (iii) organization of woreda or kebele level interface meetings between citizen and service providers so as to arrive at a joint action plan to address the identified problems/constraints; (iv) implementation of the joint action plan; and (v) monitoring improvements in access and quality of service delivery by local Social Accountability Committee.

Process Evaluation of PBS-3, Final Report: An Overview of the PBS-3 Program

4

Program, Phase-2 (ESAP-2), which was launched in November 2011, plans and implements the social accountability activities10. This program is managed by a management agency (MA).

15. The GRM is included under citizen’s engagement component to provide financial and technical support to the Ethiopian Institute of the Ombudsman (EIO), and to build capacity of the existing GRM offices in the region by developing a common standard of grievance redress procedures and handling, and by imparting training to the grievance handling officers.

16. Compared to PBS-1&2, the PBS-3 program has a strong focus on strengthening public financial management (PFM) and procurement systems at woreda-level. The main activities supported by this sub-component inter alia include development and dissemination of manuals for cash management, procurement, internal audit; rolling out of IBEX to woredas; strengthening internal audit’ and enhancing audit coverage by the office of the federal auditor general (OFAG) and office of the regional auditor general (ORAG). 17. Unlike the previous two phases, PBS-3 has a strong focus on results. It has added managing for results (MforR) to Sub-Program B. The MforR component aims at (i) monitoring PBS results; (ii) strengthening multi-sector, decentralized system for gathering timely and credible data; and (iii) undertaking targeted surveys and studies that fill crucial gaps in the GoE's and DP's understanding of how PBS results can be sustainably achieved. 18. Sub-program C: The initial design of PBS-3 included the results enhancement fund (REF), and DfID allocated GBP 50.4 million for this component. The purpose of REF was to incentivize budgetary performance of the regions by instituting an annual performance prize. It was expected to follow the framework of Public Expenditure and Financial accountability (PEFA) to assess budgetary performance of the regions. However, REF could not be implemented because of several implementation challenges including the problem of defining appropriate and verifiable benchmarks, and the difficulties of managing the competing aspirations of the regions as well as ensuring fairness of REF distribution across the regions11. In 2014, REF was dropped and the funding for it withdrawn.

19. Addition of fourth sub-component under Sub-Program B: The recent restructuring of PBS-3 (July 2015) includes a fourth sub-component under Sub-Program B, namely risk and safeguard management capacity, with a budget allocation of USD 14.5 million. The main purpose of this sub-component is to boost woreda level capacity to manage environmental and social risks, and to build capacity of the implementation support agencies viz. OFAG, the Federal Ethics and Anti-Corruption Commission (FEACC), and the Federal Public Procurement and Property Administration Agency (FPPPAA)12.

2.1.3 Core principles, tests and reviews

20. The release of block grant is linked to satisfactory compliance with mutually agreed set of core principles, which is checked through tests and reviews. There were four core principles under PBS-1, which were revised in PBS-2 and referred to as SAFE principles13. The PBS-3 program considered further revision and included eight core principles, namely effectiveness, sustainability, additionality, fairness, equity, transparency, fiduciary probity, and predictability (Table 2.1).

21. According to the recent restructuring of the program (July 2015), additionality principle and tests will not be required any more. This is because the Bank’s new program ESPES, which will now provide block grants instead of PBS-3, emphasises sustained increase in per capita block grant. The, additionality tests are therefore no longer relevant.

10 The program was originally expected to close by 31 Dec. 2015. In the steering committee meeting held on 28 Aug 2014, it was agreed to extend the completion period up to 30 June 2016. The decision was formally endorsed through Amendment No. 4 to Grant Agreement duly signed by the World Bank and the management agency on 23 March 2015. 11 DfID mentioned these points. 12 For further details refer to World Bank (July 20, 2015): Restructuring Paper. 13 These principles are: sustainability in additionality (S); accountability and fairness (A); fiduciary standards (F); and effectiveness (E).


5

Table 2.1: Core principles, tests, and reviews of PBS-3

Core principles Review/tests Effectiveness The effectiveness of service delivery is assessed through a review of the following: (i)

adequacy of medium term resource allocations to basic service sectors; (ii) balance between recurrent vs. capital spending allocations, and salary vs. non-salary expenditures; and (iii) results achieved. In every JRIS, effectiveness review is conducted for two of the five basic service sectors based on data and information gathered through pre-JRIS field missions; and the inputs provided by sector experts, sector Ministries, and DPs.

Sustainability The sustainability review considers (i) financing sources for decentralized basic services, including the government’s own revenue, contributions of DPs (through Channels 1&2), community contributions, etc.; and (ii) the unit costs of providing basic services, both current and projected costs.

Additionality This principle ensures that (a) GoE priority for MDGs are expressed in medium-term commitments to increase overall financing for the federal block grants and (b) allocated resources for basic services are flowing in a predictable manner. Two additionality tests are conducted viz. (i) non-declining trend in the proportion of allocated federal block grants (FBG) to regions in total federal expenditure (through treasury sources only i.e. excluding debt payments and MDG support fund) in a given year as well as a positive trend in the medium term; and (ii) non-declining trend in FBG allocation to regions in real terms in a given year. These two tests are conducted twice in a year as explained below: • Ex-ante additionality test (April/May JRIS): The two tests are conducted for the

next year (t+1) compared to the current year (t) • Ex-post additionality test (October/November JRIS): The two tests are conducted

for the current year (t) compared to the previous year (t-1) Fairness The objective of fairness principle is to ensure (a) that resource allocations from the

federal government to the regions, and from region to woredas, are rules-based and transparent, and (b) that block grant disbursements to the regions, as well as from regions to woredas, are executed as planned in the budget allocations. Two fairness tests are conducted in a year as given under: • Fairness test (April/May JRIS): This is an informal test and it is carried out to flag

any major discrepancies between half-yearly allocations and transfers of federal block grants (a) from federal to regions, and (b) from regions to woredas.

• Fairness test (October/November JRIS): This is a formal test and it is conducted to examine whether the allocated block grant amounts in the preceding fiscal year have been fully disbursed. For the purpose of the test, fully disbursed means that at least 90 per cent of the allocated funds have been transferred. Any disbursement less than 90 per cent needs an explanation.

Equity This principle refers to equitable access to basic services between different regions and within regions, while also accounting for gender-related differences. The equity aspect is checked through an equity review, conducted half yearly as follows: • Equity review (April/May JRIS): This review makes an assessment of the

discrepancies in access to basic services between the big and emerging regions, as well as discrepancies in terms of gender. The ‘managing for results’ component of the program is expected to provide region and gender-specific data.

• Equity review (October/November JRIS): This review is carried out based on the feedback obtained through discussions with the relevant stakeholders (e.g. woreda council members, service providers, beneficiaries) as part of the pre-JRIS field missions. The discussions with the stakeholders are conducted as per agreed terms of reference containing guiding questions.

Transparency The core tasks under FTA are dissemination of information on budget, expenditure, and service delivery; and conducting budget literacy trainings for citizens, sector officials, council members, and the PFM team members. The progress on these and related tasks are discussed in each JRIS based on pre-mission field reports, MoFED’s compiled report, and the critical comments from the regional participants. Following the


6

Core principles Review/tests discussion, the FTA actions and deliverable for the next JRIS are decided.

Fiduciary probity The PBS program requires the government to submit regular financial reports and audits to DPs to demonstrate that fiduciary standards are being maintained and strengthened. Each JRIS hears an update on progress achieved and then defines a series of deliverables for the following JRIS.

Predictability The predictability principle seeks to ensure mutual accountability by predictable resource flows for basic service delivery results. Development Partner contributions need to be based on longer-term commitments, so that agreed disbursements can be made on time. Likewise, the Government has a responsibility to accurately reflect these DP contributions for PBS in yearly government budgets. For ensuring predictability, the government and DPs are required to review all these at each JBAR.

2.1.4 Program implementation arrangement

22. Implementing agencies: The Ministry of Finance and Economic Development (MoFED) is the agency for program implementation at the federal level with responsibility for funds disbursement and smooth functioning of public financial management systems. The other implementing agencies at the federal level include: line ministries engaged in delivering basic services viz. Ministry of Education (MoE), Ministry of Health (MoH), Ministry of Agriculture (MoA), Ministry of Water, Irrigation and Electricity (MoWIE), Ethiopian Road and Transport Authority (ERTA); the Central Statistical Agency (CSA); OFAG; FEACC; and FPPPAA.

23. At the regional level, the Bureaus of Finance and Economic Development (BoFEDs) have similar responsibilities as the MoFED has at the federal level. At local level, Woreda Office of Finance and Economic Development (WoFEDs) and Urban Administration Offices of Finance have similar responsibilities as those of the BoFEDs.

24. The councils at region, zone, woreda, and kebele levels are responsible for (a) providing general oversight of program implementation; (b) reviewing and approving annual development plans and budgets; and (c) facilitating information sharing and harness the involvement of citizens in the planning, budgeting, and management of delivering basic services. They are also responsible for sustained monitoring of transparency and accountability initiatives.

25. PBS Secretariat: This was established at the outset of PBS-I and was further elaborated and formalized at the beginning of PBS-II in 2009. Under the PBS-3 program, a new ToR was developed, which defined three strategic objectives for the secretariat, namely (i) sustain and improve all partners’ understanding on the effectiveness, efficiency, and management of block grant resources; (ii) contribute to an improved understanding of the PBS program’s sector impact; and (iii) provide effective administration and logistical coordination in support of improved dialogue and supervision. The ToR also specified the scope of work under eight broader heads, and provided an organogram14.

26. However, in view of the realignment of the donors around the PBS program, the PBS secretariat has been reconstituted as Service Delivery Secretariat (SDS) with effect from July 201515.

27. Results framework and M&E: The results framework of PBS-3 includes indicators for (a) higher-level objectives, (b) PDO, and (c) intermediate results. The results indicators were first revised in 2011, then in 2013, and the last revision took place in July 2015. Including the last revision, there are now four intermediate results areas, namely: (i) improved availability of qualified staff and operational resources for delivery of basic services; (ii) improved transparency and fiduciary assurance regarding the use of public funds; (iii) improved citizens’ engagement on quality of basic services delivery; (iv) strengthened M&E systems at federal, regional, and sub-national levels; and (v) improved environmental and social management capacity. The fifth results area corresponds to the new sub-component added to Sub-Program B.

14 For further details refer to the ToR available from the PBS secretariat. 15 More will be discussed about donors’ harmonization and SDS later in this report.


7

28. JRIS/JBAR: Since inception the program review process has relied on semi-annual joint review and implementation support (JRIS) mission, which also includes joint budget and aid review (JBAR). The purpose of JRIS is to review program progress against the agreed results framework; dated covenants; and other deliverables related to system strengthening as agreed in the previous JRIS. Prior to JRIS/JBAR, field visits were often organized under PBS-1&2. The same process has been followed under PBS-3. Prior to each JRIS, field missions are undertaken focusing on thematic areas. It may be further noted that JRIS trigger disbursements of block grants.

2.2 Realignment of PBS-3

2.2.1 New program ESPES

29. As mentioned in the introductory chapter, the Bank has realigned its support to basic services by introducing a new program ESPES, which covers a three-year period, from 15 September 2015 to 5 January 2019, and replaces Sub-Program A of PBS-3 that ended on 6 July 2015 with all its funding allocations being utilised. The new program will be effective from 1 October 2015 onward. The total donors’ funding support to ESPES is estimated at USD 857.20 million, including IDA funding of USD 600 million.

30. The development objective of ESPES is to “improve equitable access to basic services and strengthen accountability systems at the decentralised level”. Equity is the key focus of ESPES and that makes it distinct from the PBS series. Another unique feature of ESPES is the program’s reliance on a new financing instrument called the program for results (PforR). This instrument allows disbursement of funds upon achievement of desired results as specified through a set of disbursement-linked indicators (DLIs)16.

31. Alongside ESPES, PBS-3 will continue with Sub-Program B until it closes in January 2019. Its sole objective will be to build capacity at woreda level and improve data collection to monitor results.

2.2.2 Other restructuring

32. GoE and DPs have expressed their support to develop third phase of ESAP, which is planned to start in July 2017. A road map with medium-term priorities/activities for this program has been finalised and approved by the steering committee17. ESAP-2 is due to close by 30 June 2016, but a bridging funding (estimated at USD 4.6 million) is expected to extend the overall disbursement closing date multi-donor trust fund (MDTF) until June 201718. The six-month period of bridging funding will ensure a smooth transition to the third phase of ESAP. 33. It is understood that a separate PFM program is being developed as a follow up of recently closed public sector capacity building program (PSCAP)19. The document is not made public and so details are not known.

2.3 Summary

An overview of the PBS-3 program

An overview of the PBS-3 program: The PBS-3 program is designed to contribute to the higher-level objective of expanding access and improving the quality of basic services by funding block grants that ensure adequate staffing and operations, and by strengthening the capacity, transparency, accountability and financial management of regional governments and authorities at woreda level. The key features of the program are as follows:

• PBS-3 consists of Sub-Program-A (block grants), and Sub-Program-B. The latter consists of three interrelated

16 For further details see PAD (August 19, 2015): Enhancing Shared Prosperity through Equitable Services 17 See ESAP, Medium-Term Priorities and Roadmap (Final Draft Version) 18 ESAP-2: Concept Note for Bridging Amendment 5, p 16-17 19 PAD (August 19, 2015): ESPES, p 5


8

An overview of the PBS-3 program

components viz. (a) citizen’s engagement (financial transparency and accountability; social accountability; grievance redress mechanism; and risk and safeguard management capacity); (b) public financial management and procurement; and (c) managing for results.

• The program rests on eight core principles (effectiveness, sustainability, additionality, fairness, equity, transparency, fiduciary probity, and predictability) of which additionality test is now dropped.

• MoFED/BoFED/WoFED, the line ministries engaged in basic service delivery, and other federal agencies implement the program, while implementation of social accountability activities (ESAP-2) is organized and managed by a separate agency. The PBS Secretariat, now reconstituted as Service Delivery Secretary, is responsible for organising JRIS/JBAR, and donors’ harmonization.

• The results framework has been revised thrice since inception. The recent restructuring has added a new intermediary results area viz. improved environmental and social management capacity. JRIS/ JBAR remains the main forum for reviewing progress and challenges of the program.

Realignment of PBS-3: Bank has realigned its support to basic services with a new program ESPES (Sept. 2015 – 5 Jan 2019), which replaces Sub-Program A. The development objective of ESPES is to improve equitable access to basic services and strengthen accountability systems at the decentralised level. The program is built on a new financing instrument PforR. The PBS-3 program will continue with Sub-Program B until it closes in January 2019.

ESAP-2 is going to enter into its third phase, for which a road map has been identified. There is also an indication that a separate PFM program is being developed as a follow up of recently closed PSCAP.

Chapter 3: Approach and Methodology

This chapter outlines the approach and methodology followed in the process evaluation of PBS-3. As suggested in the ToR, the process evaluation has been carried out applying OECD-DAC criteria.

3.1 Evaluation Framework

34. The process evaluation of PBS-3 has been carried out applying five criteria, namely relevance, effectiveness, efficiency, impact, sustainability, and coherence. The first five criteria represent aid effectiveness criteria recommended by the Development Assistance Committee (DAC) of the Organization for Economic Cooperation and Development (OECD). Given the nature of evaluation, lesser emphasis is given on impact assessment. Furthermore, assessment against the OECD-DAC criteria has been carried out using a set of evaluation questions (EQs), developed in line with the questions given in the ToR (Table 3.1).

Table 3.1: Evaluation framework and key questions

Criteria Key evaluation questions &

methodological approach Data collection

methods

Relevance

EQ1: To what extent are the development objectives of PBS-3 valid and appropriate in view of the political, economic, and social context of the country?

- Review of documents

- KIIs EQ2: How relevant are the activities and outputs of PBS-3 in achieving the stated objectives of the program?

EQ3: How appropriate are the ‘core principles’ of PBS-3?

Effectiveness

EQ4: To what extent project development objectives of PBS-3 are achieved, or likely to be achieved?


- Field mission (KIIs/FGDs)

EQ5: To what extent the PBS-3 has achieved aid effectiveness?

EQ6: What has been the role of the PBS Secretariat in improving donors’ alignment and harmonisation?

Efficiency

EQ7: How cost efficient and cost effective has been the PBS-3 program? Does it offer value-for-money (VfM)?


- KIIs EQ8: How efficient has been the program implementation?

EQ9: To what extent program risks have been efficiently managed, both in terms of identification of risks and risk mitigation measures?

Impact EQ10: What have been the impacts of PBS program till date of Phase-3, directly or indirectly, intended or unintended?


Sustainability EQ11: How sustainable is the PBS program without any funding support from the donors?


- KIIs

Coherence EQ 12: Is the program coherent with the national policy of decentralised service delivery and the sector-specific programs?


- KIIs

3.2 Methodology

3.2.1 Desk research

35. As per ToR, the present evaluation is expected to rely heavily on the secondary research. Accordingly, a large number of documents (reports, studies, aide memoire, etc.) were collected from the PBS secretariat, development partners, (World Bank, DfID, AfDB), MoFED, and other line

Process Evaluation of PBS-3, Final Report: Approach and Methodology

10

ministries. The documents were thoroughly reviewed and analyzed. In addition, a number of other reports and studies have been referred for the present study. Bibliography at the end of this report contains select readings reviewed for present evaluation.

3.2.2 Federal level interviews

The key informant interviews (KIIs) at federal level were conducted with the concerned staff of the Bank, PBS secretariat, DfID, European Union (EU), Austrian Development Corporation (ADC), and Italy Development Cooperation (IDC). The African Development Bank (AfDB) responded to specific questions through mail. A separate discussion was held with the project Task Team Lead of the Bank. The evaluation team also consulted top officials and staff of MoFED. Meetings were also held with senior officials of EMIS & ICT Directorate in the Ministry of Education (MoE), Ministry of Health (MoH), Ministry of Water, Irrigation and Electricity (MoWIE), and Central Statistical Agency. The KIIs were conducted with a semi-structured questionnaire. The list of people met and interviewed at federal level is given in Annex-2.

3.2.3 Field mission in select regions and woredas

36. Sample regions and woredas: A small-scale field mission was undertaken to get a feel about the basic services at woreda level. Given that the assignment provided for a small team of experts and limited time to complete the evaluation, it was decided to visit four regions; eight woreda, two woredas from each region; and eight kebeles, one in each woreda. The evaluation team intended to visit both developed and emerging regions, and accordingly selected Oromiya, SNNPR, Benishangul Gumuz, and Somali. The selection of regions, woredas, and kebeles was guided by the prior experience of the mission team members, and the convenience of logistics and travel time (Table 3.2).

Table 3.2: Sample regions and woredas

Region Woreda Remarks

SNNPR Shebedino SAIP woreda Dila zuria Non- SAIP woreda

Oromiya Jima arjo SAIP woreda Cheliya Non- SAIP woreda

Somali Shinile SAIP woreda Jijiga Zuria Non- SAIP woreda

Benishangul Gumuz Bambasi SAIP woreda Assosa zuria Non- SAIP woreda

37. Data collection at regional/woreda/kebele levels: The first field visit was organized in Sabeta Hawas Woreda in Oromia region on 11 June 2015 for a deeper understanding of the woreda level delivery of basic services, as well as for pre-testing of the survey instruments. The actual fieldwork started on 29 June 2015 going up to 16 July 2015. The evaluation experts were grouped into two mission teams, and each team covered two regions. In course of data check and validation, some data gaps were noticed for two regions, namely Oromiya and Benishangul Gumuz. The team members therefore re-visited those two regions during 25 September to 1 October 2015.

38. In each selected region, group interviews were conducted with the concerned staff of BoFED and heads, or representatives, of the five sector bureaus using a structured questionnaire. Part-1 of the questionnaire was structured to collect mainly financial and physical data, while Part-2 of the same questionnaire included only qualitative questions20. The Part-1 was sent to BoFED head prior to actual visit to the region so as to save on data collection time. In a similar fashion, the mission teams conducted interviews, using a two-part questionnaire, with the staff of woreda administration, WoFED, and sector bureaus. The Part-1 of the questionnaire to collect financial and physical data was administered at the time of field visit.

20 The inception report contains the full set of questionnaires and FGD checklists.

Process Evaluation of PBS-3, Final Report: Approach and Methodology

11

39. The community feedback on service delivery was gathered at kebele level through focus group discussions (FGDs). The FGD checklist contained discussion questions regarding basic service delivery. It also included questions related to social accountability activities. In all, 188 community members participated in FGDs across eight selected kebeles.

3.2.4 Data analysis

40. The field data were analyzed using standard analytical tools and graphic presentations. Besides, the present study has undertaken extensive analysis of secondary data, including counterfactual analysis to track progress towards MDGs.

3.3 Limitations of the Study

41. The present study suffers from a few limitations:

• The timeline for process evaluation of a complex program like PBS has been quite short, which restricted the scope for more detailed analysis.

• It is expected that all relevant documents and data should be available on website of the PBS secretariat. However, the secretariat has not developed its own website. As a result, a great deal of time was spent only in collecting documents and data. Some of the reports mentioned in the aide memoire could not be collected. This has limited the scope of in-depth analysis in some areas.

• A process evaluation requires a good sense of actual implementation of the activities at the grass root level. But, adequate time and resources are not allocated for that purpose. The present study has undertaken only a limited field mission; the findings can at best be treated as indicative and not definitive.

42. Notwithstanding the above limitations, it is expected that the findings and recommendations of the present study would be useful to the government, DPs, and other stakeholders.

3.4 Summary

Approach & Methodology

The present evaluation has applied six criteria (relevance, effectiveness, efficiency, impact, sustainability, and coherence) and a set of evaluation questions. Given the nature of evaluation, lesser emphasis is given on impact assessment.

The evaluation has relied on desk research, key informant interviews at federal level, and field mission in four regions (Oromiya, SNNPR, Benishangul-Gumuz, and Somali), and eight woredas. The data were collected at federal/regional/woreda level using semi-structured and structured questionnaires. The structured questionnaires were designed judicially blending quantitative and qualitative questions. The FGDs were conducted using a checklist in eight kebels and188 community members participated.

The first field mission took place from 29 June to 16 July 2015. Recognizing some data gap, field mission was repeated in Oromiya and Benishangul Gumuz during 25 September to 1 October 2015.

The present study suffers from a few limitations, such as time constraint, non-availability of all the documents, and limited fieldwork with a small sample size.

Chapter 4: Relevance and Coherence of the PBS-3 Program

In this chapter, the relevance and coherence of the PBS-3 program has been evaluated following the relevant evaluation questions mentioned in the previous chapter. The evaluation is based on extensive review of available documents and key informant interviews.

4.1 Country Context at Appraisal

4.1.1 Political development

43. The political context of the PBS-3 program at appraisal was marked by (i) long-term stability of the government albeit with some concerns; (ii) smooth constitutional transition of power following the death of the former Prime Minister; and (iii) an allegation of PBS program’s linkage to human rights abuse in Gambella region leading to heightened international scrutiny.

44. Political stability: The Ethiopian People’s Revolutionary Democratic Front (EPRDF), the present ruling party, came to power in 1991 after almost two decades of armed struggle, overthrowing the previous military Dergue regime. In the first multi-party parliamentary election held in 2005, the opposition groups lost heavily to EPRDF. Since then the country has witnessed political stability, with EPRDF winning successive elections, including the recent one held in May 2015.

45. However, the development partners and other stakeholders, including the opposition and the media, have been concerned about the limited space for citizens’ formal political engagement in the country21. The PBS donors have also been concerned with the government’s law that civil society organisation (CSO) cannot spend more than 30 per cent on administrative costs. It is understood that such restrictions on administrative costs, which supports activities such as transport, training, M&E and supervision, might limit CSO operations22. However, PBS-3 has not been affected by any of these concerns since DPs have been regularly in dialogue with the government to discuss and sort out any issues related to program implementation. Regarding CSO law, the government assured through a letter in February 2009 that the law would not impede implementation of social accountability activities of the PBS program23. Till date, GoE has honoured this commitment, and the social accountability activities have progressed well.

46. Constitution transition of power: In 2012, H.E. Haile-Mariam Desalegn was appointed as Prime Minister of the country, following the death of the former Prime Minister, Meles Zenawi. This was the first peaceful and constitutional transition of power in the political history of Ethiopia. For the development partners, this was a new milestone of political stability.

47. Heightened international scrutiny: The PBS program came under the scanner of international scrutiny right at the inception stage of PBS-3. The events leading to this situation are as follows24.

48. On 24 September 2012, the World Bank’s Inspection Panel received a request for inspection related to PBS-II (Additional financing) and PBS-III, wherein the requesters alleged that the Bank supported PBS program harmed the Anuak people in Gambella region. The harms described in the request covered four aspects: (i) the government’s takeover of people's customary land under community development plan (CDP), also known as villigisation program, without their free and informed consent; (ii) use of force and intimidation to get people to move; (iii) lack of public services and assistance at relocation site; and (iv) deterioration of livelihood due to less fertile land and lack of other income generating opportunities at relocation site. The Panel’s independent investigation report (21 November 2014) concluded that the PBS program did not cause alleged harms. However, the Panel observed that the PBS-3 appraisal and supervision did not pay adequate attention to mitigate the

21 See PAD (29 August, 2012): PBS III, Annex 6, p 96. The same concern was also echoed by AfDB; see Appraisal Report (July 2012): PBS III, Para 2.2.1, p 3. 22 PAD (29 August, 2012): PBS III, Annex 6, p 96. 23 Ibid, p29 24 For more details, see the Inspection Panel Report and Recommendations, 8 Feb. 2013.

Process Evaluation of PBS-3, Final Report: Relevance and Coherence of the PBS-3 Program

13

potential risks arising from geographical and sector overlap of PBS and CDP. The Bank responded to the concerns raised by the Panel with a comprehensive action plan. This is discussed later in this report (see Sec. 7.3, Chap. 7).

4.1.2 Economic development

49. The economic and fiscal developments around the start of PBS-3 were impressive. The GDP growth averaged 11.1 per cent between 2004-05 and 2010-11, and then moderated through the years to reach 10.3 per cent in 2013-14. The tax revenue increased from ETB 14.2 billion in 2005-06 to ETB 133.1 billon in 2013-14 representing an average growth rate of 32.3 per cent. Correspondingly, tax-to-GDP ratio went up from 10.8 per cent to 12.7 per cent. The headline inflation dropped by four per cent points, from 12.3 per cent in 2005-06 to 8.1 per cent in 2013-14, and thereby supported achieving certain level of macroeconomic stability25. Fiscal deficit (excluding grants) declined from 7.4 per cent in 2005-06 to 3.8 per cent in 2013-14 on the back of strong economic growth and phenomenal increase in tax revenue. Despite greater financing requirements of GTP and consequent dependence on external financing, debt-to-GDP ratio of the country stood at 27.9 per cent in 2013-14, indicating a low risk of external debt stress.

4.1.3 Progress towards MDGs

50. At the time of PBS-3 appraisal, Ethiopia had made impressive progress towards MDGs. The country witnessed (a) decline in poverty headcounts from 44.2 per cent in 1999-00 to 27.8 per cent in 2011-12; (b) reduction in underweight children under 5 years of age reduced from 47.2 per cent in 2000 to 28.7 per cent in 2011-12; and (c) achievement of MDG target of reducing under five mortality rate (U5MR) two years ahead of the timeline26. Furthermore, the country was on track for achieving MDGs related to universal primary education, gender parity in education, HIV/AIDS, and malaria.

4.1.4 Recurrent challenges of basic service delivery

51. Notwithstanding the impressive scaling up and improvements in basic service delivery, PBS-3 at appraisal confronted several recurrent challenges and constraints, including the following (i) shortage of skilled or trained human resource; (ii) high staff turnover; (iii) budget constraint for operating expenditure and capital investment. These problems and still impeding PBS-3 implementation. The other perennial issues were lack of service quality, and persistent inequities in service delivery across the regions and income/wealth quintiles (see Annex 4 for evidence of inequities in education and health sectors).

4.2 Program Relevance

4.2.1 Validity of program development objective

52. Since inception, the PBS program has remained a relevant program in the strategic context of the country. The reasons are not far to seek. First, PBS has remained embedded in the GoE’s policy of providing block grants to support decentralised delivery of basic services at region/woreda levels. Second, the PBS program has been well aligned with the GTP and its MDG targets. The development objective of the PBS-3 program embraces four out of seven pillars of GTP27. Third, despite significant expansion and improvements in basic services, the woredas have been grappling with problems such as resource constraints, staff shortage and turnover, capacity constraints, and weak transparency and

25 Inflation spiked to as high as 36.4 per cent in 2008-09 and 34.1 per cent in 2011-12. 26 See MoFED (2012): Assessing progress towards the Millennium Development Goals. Also see Chap. 8 27 The program development objective of the PBS program is well aligned with the fifth and sixth pillars of the GTP viz. (a) enhancing expansion and quality of social development (fifth pillar), which inter alia aims at improving education, primary health care, and safe water and sanitation; and (b) strengthening good governance and capacity building (sixth pillar). The program’s support to agriculture and road sectors are linked with the second and fourth pillar of GTP, namely maintaining agriculture as major source of economic growth, and enhancing expansion and ensuring quality of infrastructure development. Thus, the development objectives of the PBS program embrace four out of seven pillars of GTP.


14

accountability systems. The PBS program has been supporting GoE in addressing some of these challenges by supporting woreda-focussed capacity building and systems strengthening.

53. All the aforesaid points remained valid at appraisal of PBS-3, reassuring the relevance of the program objectives in the country context. However, the present evaluation finds that the PBS-3 has not addressed adequately the issue of equitable services although the evidence of inequities in sectors like basic service sectors was quite apparent at the appraisal stage. Though the results framework of PBS-3 demands region-wise disaggregated data for high-level and PDO level indicators, it has not included any specific indicators to assess progress towards bridging regional disparities and income inequities in accessing basic services. The Bank’s new program ESPES now fills this gap and aims at providing benefits of basic services to the entire population, particularly the bottom 20 per cent of income groups, and most poorly performing woredas.

4.2.2 Relevance of PBS-3 activities and outputs in achieving PDO

54. Theory of change for improving basic services: It is imperative to develop a theory of change to assess the relevance of activities and outputs in achieving the stated objectives of a program. Following this imperative, a plausible theory of change has been developed to explain how programs and interventions are expected to protect or promote basic services (Figure 4.1). The framework clearly brings out that the PBS program, with its financial contribution to block grants and initiatives to strengthen local governance, is not the only way to protect/promote basic services in the country. The basic service sectors are supported by a number of other programs.

55. For instance, the MDG fund provided by the federal government supports capital investments in basic service sectors. The universal rural road access program (URRAP) is a major source of financing for construction of rural roads. The agricultural growth project (AGP) supports small-scale investments in rural infrastructure to improve productivity.

56. Similarly, in the health sector, the MDG Performance Fund is a significant source of finance for procurement of medicines. The general education quality improvement project, second phase (GEQIP-II) provide funding support for curricular reform; teacher development program; school improvement plan; management and capacity building including EMIS; improving quality of learning and teaching through ICT; and program coordination, M&E, and communication28.

57. Referring to the theory of change explained above, it is reasonable to say that PBS-3 activities and outputs are definitely relevant for program development objective. However, it should be borne in mind that the sector specific programs also complement the PBS program. Without the support of those programs, it would not be possible to scale up basic services in all the woredas. However, for achieving desired outcomes and impact, it is imperative to integrate the PBS program with the sector specific programs, otherwise there will be duplication of interventions leading to sub-optimal use of resources. The program integration therefore should be one of the key considerations for on-going efforts towards donors’ harmonization.

4.2.3 Relevance of focusing on five sectors

58. Since block grants are allowed to finance only the recurrent expenditure (salaries and operating expenditure) of the woreda-level basic services, it will benefit those sectors that require larger fund for recurrent expenditure such as education, health, and agriculture extension services. The sectors like water and rural roads require more capital investments, which are not funded by block grants. Besides, the woredas allocate a miniscule share of block grants to these two sectors viz. 1.4 per cent for road and 2.5 per cent for water sector. Thus, PBS support to block grants has hardly benefited water and road sectors. This evaluation therefore feels that instead of thinly spreading resources, PBS-3 could have covered only education, health, and agriculture sectors.

4.2.4 Discontinuation with LIG and health components

59. LIG was designed to give pilot woredas an opportunity to use discretionary fund for capital investment, and to enable them learn capital budgeting, procurement, safeguards, and contracts

28 See the World Bank (Oct 17, 2013): General Education Quality Improvement Project II


15

management29. An independent evaluation of LIG highlighted that despite capacity constraints, the woredas implemented a large number of projects with LIG support, and in the process, demonstrated willingness to learn and adopt national standards/systems30.

Figure 4.1: Theory of change for protecting/promoting basic services

29 LIG support was provided to 51 pilot woredas under PBS-1 and to another 48 pilot woredas under PBS-2. 30 See for further details IPE Global (Nov 2011): Evaluation of Pilot Local Investment Grant

Service Delivery

• Increased number of frontline service providers (teachers, HEWs, DAs, staffs in road office/desk, etc.)

• New/up-graded/rehabilitated infrastructure & facilities

• Improved supply of equipment & commodities (drugs, text books, teaching aids, etc.)

Good Governance

• Improved citizen engagement (FTA/SA/GRM)

• Strengthened PFM and procurement systems

• Reliable multi-sector MIS & improved M&E systems

• Sub-Program-A (Block grants)

• Sub-Program-B (PFM/FTA/GRM/MforR)

Chapter 1 ESAP (off-budget)

PBS

• MDG Fund (Capital exp.)

• OWNP • URRAP • AGP • GEQIP-II • MDG Performance Fund

(Health)

Com

plem

entin

g Pr

ogra

ms

INPUT OUTPUT

• Expanded access & quality basic services • Improved woreda capacity, transparency, accountability

& financial management

OUTCOME

IMPACT

• Progress towards MDGs & GTP targets • Institutionalisation of supportive systems

Source: Developed by the evaluation team Notes: • MDG fund finances investments in basic service infrastructure; OWNP invests in WASH; URRAP

supports investments in rural roads; AGP supports small-scale investments in rural infrastructure to improve productivity; GEQIP and MDG Performance Fund (health) finances, among others, commodities.

• All sector programs include M& E components


16

60. LIG was a successful pilot project, but its scaling up to national level through PBS program was simply not a feasible and desirable option. LIG was therefore rightly dropped from the PBS program, and a new MDG Fund was launched in 2011-12 to support essential local level infrastructure investments in the basic services. However, most of the woredas do not have capacity to handle large procurement and contracts management. The MDG funds are therefore managed at the regional level. There is an urgent need to build capacity of the woredas to undertake construction works, which possibly could have been a focus area under PFM component of PBS-3.

61. The PBS-3 program also dropped the health component, and did it rightly. The health component was a sector-specific intervention and its inclusion was not a good fit for a program like PBS that is not intended to have any sector specific interventions. The discontinuation of the health component has not caused any programmatic gap since MDG Performance Fund (MDG PF) meets the funding gaps for health commodities (including medicine) mainly to support maternal and child health care services at primary health facilities31.

4.2.5 Relevance of core principles

62. Compared to earlier three phases, PBS-3 has included more core principles, including a new series of reviews pertaining to effectiveness, sustainability, and equity (see Table 2.1, Chap. 2). These core principles are of great relevance to the program, and most of them are adequately defined. However, there are implementation challenges as highlighted below:

• Sustainability review: Sustainability has always been a core concern of the PBS program. But, it has been formalised in PBS-3 through a separate review. However, the focus of this review is limited to the funding sources for decentralised services, and estimation of unit cost of basic services. Ideally, such review should have considered institutional sustainability.

• Equity review: Equity assessment is a challenge because of high government sensitivity about the definition of equity. It is understood that equity review as described in PAD (see Table 2.1 in Chap.2) has not progressed beyond discussion with the stakeholders during pre-JRIS field mission.

• Predictability: The predictability review involves checking whether the donors’ committed funds are included in the budget proclamation. It is not clear how this measures predictability. A logical approach would have been to compare actual release of funds with the donors’ commitments.

4.2.6 Appropriateness of results framework

63. The results framework of the PBS-3 program covers all the basic sectors. In recent restructuring of PBS-3 (July 2015), the framework has been expanded to include a few indicators to monitor results related to the newly added sub-component viz. risk and safeguard management capacity. However, the results framework can be strengthened further by addressing the following weaknesses:

• Inappropriate high-level health indicators: The results framework of PBS-1&2 used maternal and child mortality rates as high-level (impact) indicators, whereas PBS-3 has replaced those two indicators by two PDO level (outcome) indicators viz. ‘proportion of children vaccinated,’ and ‘pregnant women received ANC during first visit’. The present evaluation strongly feels that PBS-3 should continue with the high-level indicators like maternal and child mortality rates even if data are not tractable on a yearly basis.

• Absence of technical note to define indicator for measurement: The results framework does not provide any technical notes to define indicators for measurement. For instance, number of ‘direct beneficiaries’ is a PDO level indicator for agriculture. But, there is no technical note to explain who the direct beneficiaries are, and how their numbers are estimated (i.e. through survey or any other methods). To give another example, PBS-2 used ‘average distance to

31 The Federal Ministry of Health (FMOH) manages the MDG Performance Fund. For further details see HSDP IV, Annual Performance Report, 2013-14.


17

nearest all-weather road’ as high-level indicator. PBS-3 replaced it by ‘average time to nearest all-weather road’. The question arises how time is to be calculated (measurement definition). One may also like to know which one is a better indicator – distance or time?

• Use of dual data source: Both HMIS and DHS data have been used to define targets, and usually the numbers are hugely different. It is doubtful whether any rigorous exercise has been undertaken to set year-wise targets in a comparable way using HMIS and DHS data. The present evaluation does not see any value of using multiple data source for tracking a result indicator. In case the donors insist on using multiple data source, it is imperative to add a technical note stating the way targets are calibrated for each data source.

• Data inconsistencies: The present evaluation finds some data inconsistencies. For example, significant discrepancy has been noticed in the year-wise number of HEWs reported by the GTP progress report, April 2015 JRIS presentation (by Policy and Planning Directorate, MoH), and Aide Memoire, 27-30 April 201532. This kind of discrepancy is not unusual, but it is important to reconcile the figures, or at least mention in the results framework which data source is used and why.

4.3 Program Coherence

64. The PBS program is highly coherent with the national policy of providing block grants to support decentralised delivery of basic services at region/woreda levels. In particular, Sub-Program A is well aligned with the fifth pillar of GTP, namely enhancing expansion and quality of social development, which inter alia aims at improving education, primary health care, and safe water and sanitation. Similarly, Sub-Program B is coherent with the sixth pillar of GTP viz. strengthening good governance and capacity building.

65. Besides, as evident from the theory of change (Figure 4.1), both the sub-programs complement each other in improving basic service delivery. In fact, Sub-Program A cannot be implemented effectively without Sub-Program B. For efficient and effective spending of block grants, it is important to improve local funds management, transparency and accountability, and deep citizen engagement in all facets of service delivery. There is also a growing realization that without strengthening local governance institutions and systems, sustained delivery of quality basic services is simply not feasible.

66. But, a mooted question is what should have been the scope of coverage of institutional reforms under Sub-Program B. It seems may that Sub-Program B has embraced too many challenging areas. The program could have focused only on a few priority areas. For instance, under PFM reform the focus could have been on building procurement capacity, if that is being identified as the weakest performance area at woreda level. Similarly, under ESAP the first agenda could have been to sensitize the local administration and service providers about the citizen’s right to demand and their obligations to provide basic services, and strengthen capacity of the existing community based organisations (e.g. parent teachers association at school; health boards; water committees) so as to enable them to work with local administration service providers for improving basic services. The main argument is that one sub-program cannot simultaneously address, how well it might have been packaged, all the age-old issues of funds management, transparency and accountability, and citizen engagement. The present evaluation is of the opinion that Sub-Program B has been over designed.

67. The theory of change also highlights strong complementarity between PBS and the sector programs. While PBS is supporting recurring expenditure, sector programs along with MDG funds are taking care of infrastructure and facilities as well as supply of commodities. Harmonization of all these programs would be key to enhance quality service delivery.

32 The source-wise data discrepancies are shown in a footnote table in Chap.5.


18

4.4 Summary of Findings

Relevance and Coherence of the PBS-3 Program

Country context at appraisal: The country scenario around appraisal of PBS-3 was marked by political stability; heightened international scrutiny following the alleged association of the PBS funding and human rights abuse in Gambella region leading to intense international scrutiny of the program; encouraging economic development; and impressive advances towards MDGs. Despite significant scaling up and improvements, basic service delivery at woreda was affected by a number of recurrent challenges (shortage of skills, staff turnover, budget constraint), poor service quality, and inequities in basic service delivery (across regions and income/wealth quintiles).

Program relevance: The program development objective of PBS-3 was valid in the country context at appraisal. However, the program did not address adequately the issue of equitable services although the evidence of inequities in sectors like education and health was quite apparent at the appraisal stage. The other findings are as follows:

• The theory of change clearly shows that PBS-3 activities/outputs are relevant for achieving PDO. It also highlights complementarity between the PBS and the sector specific programs. Integration of PBS and sector programs should be one of the key considerations for on-going efforts towards donors’ harmonization.

• The water and road sectors require more capital investments, while the PBS program supports only the recurrent expenditure. Thus, PBS has hardly benefited these two capital-intensive sectors. Instead of thinly spreading resources, PBS-3 could have covered only education, health, and agriculture sectors.

• The core principles are of great relevance, but in practice sustainability and equity reviews have fallen far short of requirements. Besides, the predictability assessment does not appear to follow a logical approach. The assessment does not involve comparison of actual elease of funds with the donors’ commitments

Appropriateness of results framework: The results framework suffers from some shortcomings such as inappropriate high-level health indicators; lack of technical note to define indicator for measurement; use of double data source without setting comparable targets; and some data inconsistencies.

Program coherence: The Sub-Program-A and Sub-Program-B complements each other. For efficient and effective use of block grants, it is important to improve local funds management, transparency and accountability, and deep citizen engagement in all facets of service delivery. However, the Sub-Program A seems to be more ambitious and attempts to cover too many challenging areas. The program could have focused only on few priority areas. There is also strong complementarity between PBS and the sector programs. Harmonization of all these programs would be key to enhance quality service delivery.

Chapter 5: Program Effectiveness (Progress in Achieving Results)

The effectiveness of PBS-3 program has been assessed in terms of the extent to which the program has achieved the targets set in the results framework. Annex-5 contains detailed results and charts. The evaluation team also visited four regions (Oromiya, SNNPR, Benishangul- Gumuz, and Somali), eight woredas, and eight kebeles. The findings of the field visits are also presented in this chapter.

5.1 Effectiveness of Basic Service Delivery

5.1.1 Progress towards achieving higher-level and PDO targets

68. The progress in the basic service sectors has been assessed using official statistics (wherever possible), the figures quoted in the results framework in April 2015 Aide Memoire, and the results presented in April 2015 JRIS. The assessment covers two years, 2012-13 and 2013-14. It may be pertinent to mention here that the PBS-3 is not the only program that supports basic service sectors. There are other sector programs and initiatives that are contributing to basic services (refer to theory of change in Chap. 4). Besides, the PBS contributions are not expected to be uniform across all the basic sectors. The education, health, and agriculture sectors are likely to be benefited most from the PBS program (see Sec. 4.2.3, Chap. 4).

69. During the first two years of implementation, higher-level and PDO targets were moderately achieved (Table 5.1). Of the seven higher-level indicators, targets were achieved only for three indictors (43 per cent) in 2012-13, and for four indicators (57 per cent) in 2013-14. Similarly, of the six PDO indicators, targets were achieved for three indicators (50 per cent) and four indicators (67 per cent) in 2012-13 and 2013-14 respectively. All the higher-level and PDO targets were missed in the water sector whereas all the targets were achieved in the road sector. Both these sectors require high capital investments, and their performance has little or no bearing with the PBS program.

Table 5.1: Achievement of targets in 2012-13 & 2013-14

Sectors / indicators Higher-level

indicators PDO indicators

2012-13 2013-14 2012-13 2013-14 Education NER, Grade 5-8 (%) ● ● GPI, grade 5-8 ▲ ▲ Student-teacher ratio, Grade 5-8 ▲ ▲ Qualified teachers (%) ● ● Health (high PBS contribution) Penta-3 vaccine coverage ● ● ANC during first visit ▲ ▲ Population per HEW ● ▲ Agriculture Major food crop productivities ● ▲ Direct beneficiaries of extension services ▲ ▲ Water & sanitation Rural water supply coverage ● ● Non-functional water schemes ● ● Rural roads Time to nearest all-weather road ▲ ▲ Roads in fair and good conditions ▲ ▲ ▲Target achieved ● Target not achieved

20

Process Evaluation of PBS-3, Final Report: Program Effectiveness (Progress in Achieving Results)

5.1.2 Sector wise progress performance

70. Education: Net enrolment ratio (NER) at Grade 5-8 dipped from 48.1 per cent in 2011-12 to 47.2 per cent in 2012-13, and then marginally increased to 49.5 per cent in 2013-14. In both the years, NER was significantly below the targets as set in the results framework. A consistently lower NER at Grade 5-8 is a matter of great concern since it reflects dropouts of children in transition from lower to upper primary level33. Notwithstanding low NER, the gender parity index (GPI) at Grade 5-8 has been hovering around one quite for some years, indicating a satisfactory progress towards girls’ enrolment in primary education in the country.

71. In recent years, pupil-teacher ratio (PTR) at Grade 5-8 has remained far below the norm of 50 students per teacher. The ratio stood at 44.2 in 2010-11, and it further declined to 36.4 in 2013-14. Such low PTR implies underutilisation of the existing teachers. In contrast, PTR at Grade 1-4 stood at 59.4 in 2013-14, much above the norm. This means that deployment of teachers at lower primary level has not kept pace with the steep rise in NER (107.9 per cent in 2013-14), and there is an urgent need to deploy more teachers at lower primary level.

72. The proportion of qualified primary teachers (diploma and above) at Grade 1-8 was 64.7 per cent and 70.2 per cent in 2012-13 and 2013-14 respectively. In both the years, availability of qualified teachers remained significantly below the targets. However, disaggregated data shows that in 2013-14 54.7 per cent of lower primary teachers were qualified compared to 91.7 per cent qualified upper primary teachers. This means lack of qualified teachers is a problem only at Grade 1-4, and not at Grade 5-8. So, more attention is required for increasing qualified teachers at Grade 1-4.

73. The PBS program has not considered learning achievement as a higher-level outcome although it is a proximate indicator to assess quality of education. Among others, quality of education depends to a large extent on the availability of competent and motivated teachers. According to a small survey conducted in May 2014, most of the schoolteachers lack pedagogical skills and proficiency in teaching English34. The same survey also reveals that the teacher absenteeism is not high (12 per cent), but proportion of teachers not taking scheduled classes is quite high (28 per cent). Thus, teacher motivation is a challenge.

74. Health: The Penta-3 vaccine coverage increased from 84.9 per cent in 2011-12 to 87.6 per cent in 2012-13 and then to 91.1 per cent in 2013-14. Despite this increase, the vaccine coverage remained below the targets set for 2012-13 and 2013-14. The ANC coverage for pregnant women declined marginally from 2.86 million in 2012-13 to 2.52 million in 2013-14, but the coverage exceeded the targets in both the years.

75. The population per HEW was below the norm of 2,500 in 2011-12 and 2012-13. But, the position improved in 2013-14 with population per HEW reaching the norm. According to latest statistics, cumulative number of HEWs has reached to 42.3 thousand, covering 2,129 populations per HEW35. However, the present evaluation is not confident about reliability of the reported number of HEWs due to discrepancies observed across different sources of data36.

33 There are several socio-economic reasons for high dropouts. These are discussed in Annex-6. 34 For details see Wendmsyamregne Mekasha Zike and Bereket Kelemu Ayele (2015): “State of Service Delivery in Ethiopian Primary Schools: Findings from the Ethiopian Education Service Delivery Indicator Survey”, Middle Eastern & African Journal of Educational Research, No 16, 34-58. 35 Health and Health Related Indicator, EFY 2007 36 The table below shows discrepancies in the reported number of HEWs across different sources.

Nos. GTP Progress Report, 2013-14 April 2015 JRIS presentation* Aide Memoire, April 2015 2010-11 38,298 30,948 38,298 2011-12 38,102 31,135 36,714 2012-13 36,336 34,850 38,003

* Presentation by Policy and Planning Directorate, FMoH

21


76. As noted in the previous chapter, the results framework of PBS-3 has not set any higher-level indicators to track child and maternal mortality, even though health MDGs (Goals 4 & 5) include these two indicators. While IMR has declined considerably, and U5MR target has been achieved ahead of time, MMR stands at very high level, 676 deaths per 100,000 live births as of 2011. There is an indication that MMR has declined to 420 deaths per 100,000 live births in 2013. Even with this figure, the country remains off-track in achieving MDG target of 267 deaths per 100,000 live births37.

77. The presence of skilled health personnel at birth is one of the proven interventions to reduce maternal mortality. In Ethiopia, percentage of institutional deliveries assisted by skilled professionals has remained extremely low viz. 10.0 per cent in 2011, and 14.5 per cent in 201438. Such low level of institutional delivery inter alia underpins the common belief among many women that institutional delivery is ‘unnecessary’ and ‘not a customary practice’39. Among the African countries, Ethiopia, Niger and Sudan have the least access to skilled health attendants40.

78. According to SPA+, most of the health posts (above 90 per cent) are providing ANC, family planning, and curative services for children less than five year. However, normal delivery service is available only in 45 per cent health posts41. Most of the health posts are also faced with short supply of essential medicines.

79. Agriculture: The productivity of major food crops (cereals, pulses, and oil seeds) increased from 17 quintals per hectare in 2011-12 to 17.8 quintals per hectare in 2012-13, and 20.3 quintals per hectare in 2013-14. The productivity target was missed in 2012-13 by 2.4 quintals per hectare, but then production increased and almost touched the target in 2013-14.

80. The main focus of agriculture extension services is to promote utilization of improved agriculture technologies and natural resource conservation and management. Between 2011-12 and 2013-14, direct beneficiaries of agriculture extension services (smallholder farmers and pastoralists) increased from 10.51 million to 13.09 million, achieving the national targets.

81. The number of DA with specialization in agro-ecological zones and high value crops was 2,170 and 2,811 in 2012-13 and 2013-14 respectively. In both the years, the numbers of DA were below the targets. However, The present evaluation is not sure about the reliability of the reported number of DAs since there is no published document.

37 Using proxy indicators such as skilled birth attendance and contraceptive rates, WHO has estimated MMR at 420 deaths per 100,000 live births as of 2013. The EDHS estimate of MMR will be available when CSA comes out with its publication in 2016-17. 38 These figures are quoted from EDHS 2014. According to HMIS, percentage of deliveries assisted by skilled health attendants are as given in the table below. However, HMIS figures are not comparable with the EDHS estimates, which are survey data gathered through recall period up to five year prior to survey. HMIS estimates are based on supply data for the same year.

(%) 2010-11 2011-12 2012-13 2013-14 Deliveries attended by skilled professionals 17 20 23 41* * The achievement is significantly below the target of 60 per cent Source: HSDP IV, Annual Performance Report, EFY 2006

39 UNFPA (Dec.2012): Trends in Maternal Health in Ethiopia - Challenges in Achieving the MDG for Maternal Mortality 40 United Nations: MDG Report 2014: Assessing Progress in Africa toward the Millennium Development Goals 41 It may be noted that 3.9 per cent of the deliveries are conducted by HEWs. See Health and Health Related Indicator, EFY 2007, Table 3.1.2, p 18.

22


82. It is understood that DA retention and quality of service have been a regular problem due in large part to poor living and working conditions, and inconsistent implementation of career paths and incentive packages42. PBS-3 needs to address these problems.

83. Water and sanitation: The rural water supply coverage increased from 55.2 per cent in 2011-12 to 66.5 per cent in 2012-13, and 75.5 per cent in 2013-14. Despite this increase, water supply coverage fell short of targets in 2012-13 and 2013-14. An option for improving water supply coverage is to reduce non-functional water schemes. Though non-functional water schemes declined from 24 per cent in 2011-12 to 15.5 per cent in 2013-14, the targets were not achieved.

84. In the absence of reliable time series data on use of latrines by the households, the present evaluation has relied on Ethiopia Demographic and Health Survey (EDHS) data. The households using some form of improved or non-improved facility increased significantly, from 61.7 per cent in 2011 to 66.5 per cent in 2013-14. At present, only 26 per cent of kebeles have achieved the status of open defecation free (ODF)43.

85. Rural roads: The rural road connectivity has improved significantly over the years. In 2009-10, there was no all-weather woreda road in the regions. Since then a total of 39 thousand km of woreda road was constructed under URRAP up to 2013-14, covering 68 per cent of the existing kebeles44. Following the aforesaid construction, the average travel time to reach nearest all-weather road decreased from 2.9 hour in 2011-12 to 2.1 hour in 2012-13, 1.8 hour in 2013-14. The year-wise reductions in travel time were very close to the targets.

86. According to official statistics, 86 per cent of the roads were maintained in fair and good conditions throughout the period from 2011-12 to 2013-14, which exceeded the targets set in the results framework. Furthermore, since 2011-12 the road sector has established and maintained 500 woreda desk offices with appropriate staff to oversee construction and maintenance of rural roads.

5.2 Mission Findings on Basic Service Delivery

5.2.1 Key findings based on staff interviews at select regions/woredas

87. Service standards: Except two select woredas in Benishangul-Gumuz, the other six woredas have developed service standards for education and health services. This is an encouraging trend. However, developing service standards is a necessary but not a sufficient to improve service quality. There must be an effective M&E system to ensure proper implementation of the service standards. The present evaluation could not probe this aspect due to time constraint.

88. Facility survey: All the regions barring Benishangul-Gumuz have claimed that they conduct survey of school, health, and water supply facilities. For instance, BoFED and the Bureau of Education have conducted facility assessment of school and other educational institutions. Some of the woredas have also conducted some sort of facility survey. All these initiatives are important for preparing a proper inventory of infrastructure/facilities and in identifying the gaps that are impeding service delivery. At present, this is a weak link in the basic service delivery.

89. Beneficiary survey: None of the regions and woredas visited conducts beneficiary survey to take feedback on service delivery. Neither they are systematically gathering feedback from any other sources. This is not a good practice. Going forward, beneficiary feedback must be an integral component of the existing MIS of the service bureaus.

42 See Ethiopian Agricultural Transformation Agency, Annual Report, 2013-14, p 68. Also refer to PAD (29 Aug 2012): Promoting Basic Services Phase III, p 5-6. 43 Health and Health Related Indicator, EFY 2007, p 47 44 The quoted figures are obtained from the presentation at April 2015 JRIS. To achieve the five-year construction target of 71.5 thousand km under URRAP (see GTP Annual Progress Report 2014, Table 22), about 33 thousand km should have been constructed during 2014-15. It is not known whether this target has been achieved.

23


90. Woreda level performance in service delivery: All the regions visited have an understanding about the performance of woredas in basic service delivery. When enquired about what proportion of woredas are providing basic services reasonably well, the regions could provide some indications. It appears from the regional responses that the proportion of woredas that are providing better services is more than 75 per cent in SNNPR; 50-75 per cent both in Oromiya and Somali; and 25-50 per cent in Benishangul-Gumuz. The regions are not only tracking service delivery at woreda level but also taking measures to improve service delivery. Examples of such measures are as follows:

• Oromiya formed groups of experts to assess performance gaps in service delivery. Based on gap analysis, the region provided trainings to staff and the beneficiaries. Besides, the region utilised MDG, ULGDP, and others funds to improve infrastructure and facilities.

• Somali region conducted regular assessment of service delivery using agreed performance criteria. Based on the assessment results, on-the-job trainings were organised to develop capacity of the staff.

• SNNPR identified lagging woredas (e.g. Segen zone and South Omo woredas) for providing special support such as support for conducting monitoring and evaluation; provision for computers, vehicles, etc.; provision for support staff; on-the-job training for staff at zone level; and organising training on regular basis related to gap areas.

91. As evident from above, the regions have willingness to, as well as some experience of assessing service delivery at woreda level. It may be pertinent now to introduce an appropriate ‘results based, and incentivised’ system to evaluate and monitor service delivery at woreda level as per standards.

92. Performance of the service providers: Of the eight woredas visited, seven woredas (87.5 per cent) grade performance of the primary/secondary schools using some rating criteria, while a relatively lesser number of woredas (50 per cent) evaluate and grade performance of the health facilities (Table 5.2). However, none of the agriculture bureaus track delivery of extension services at kebele level. Similarly, none of the rural road bureaus (or rural desks) assess rural road services.

Table 5.2: Performance of schools and health facilities as per service bureaus

Percentage Oromiya Somali SNNPR Benishangul -Gumuz

Chelia Jima Arjo Jigjiga Shinile Dilla

Zuria Shebedino Asosa Bambasi

Proportion of primary schools performing well

Less than 25 ✓ ✓

Scho

ols n

ot

rate

d 25 to 50 ✓ ✓

51 to 75 ✓ ✓

Above 75 ✓

Proportion of health posts performing well

Less than 25

Hea

lth p

osts

no

t rat

ed

Hea

lth p

osts

no

t rat

ed

Hea

lth p

osts

no

t rat

ed

Hea

lth p

osts

no

t rat

ed

25 to 50

51 to 75 ✓

Above 75 ✓ ✓ ✓

Proportion of health centres performing well

Less than 25

Hea

lth c

entre

no

t rat

ed

Hea

lth c

entre

no

t rat

ed

Hea

lth c

entre

no

t rat

ed

Hea

lth c

entre

no

t rat

ed

25 to 50 ✓ 51 to 75 ✓ Above 75 ✓ ✓

24


93. The responses of the sample woredas further reveal that in two woredas viz. Chelia and Shinile, only a smaller proportion of primary schools (less than 25 per cent) are doing well (Table 5.2). For improving service delivery, Chelia has taken several measures such as training of school principals and staff; experience sharing with the model school on a monthly basis; regular follow up meetings; and ranking of schools. Shinile has also prepared an improvement plan but it is facing budget constraint to implement the plan. Turning to health service providers (health posts, health centres), the performance is somewhat better. According to health bureaus in Jima Argo, Dilla Zuria, and Shebedino, more than 75 per cent of their health posts are doing well. Similarly, more than 75 per cent of the health centres in Dilla Zuria and Shebedino are performing well.

94. That the sector bureaus in some of the woredas are tracking and grading the service providers is an encouraging progress towards improving service delivery. This practice needs to be strengthened and scaled up to all woredas. The regions are urgently required to introduce a standardised evaluation and grading system for each of the five basic services, including beneficiary feedback.

95. The challenges and constraints of service delivery: The regions/woredas are faced with some key challenges and constraints as summarised below:

• High staff turnover: Almost all the regions/woredas are facing the problem of staff turnover. Poor salary, limited growth opportunity, poor work environment, and heavy workload are the main reasons for staff turnover. According to one woreda head, “only the older people want to stay and work at woreda. The younger graduates want to try their luck in bigger cities like Addis Ababa”.

• Shortage of skilled/trained human resource: Most of the woredas are constrained by shortage of skilled human resource for service delivery. Several factors are responsible for staff shortage, such as high staff turnover; inadequate staff training; recruitment of staff without matching job profile; etc.

• Budget constraint: The woredas are faced with shortage of funds for office operations; repairs and maintenance works; capital investment; staff recruitment and training; public awareness campaigns; support supervision; M&E/data collection; etc.

• Logistic problem: Most of the service bureaus at woreda level do not have vehicles to reach out to the service points, especially in remote areas. For instance, in Chelia woreda (Oromiya) engineers are forced to walk long hours and many left the job mainly due to this factor.

• Sector specific problems: There is wide range of sector specific problems apart from those mentioned above. Some examples are as given under:

- Lack of local suppliers of textbooks and lab equipment (Dilla Zuria) - Inadequate health posts; shortage of equipment, transport, ambulance (Shinile) - Shortage of medicines (Dilla Zuria) - Non-motorable road to reach remote areas to provide agriculture extension activities,

and to support community during animal and crop disease outbreak (Chelia) - Shortage of agriculture inputs (Bambasi) - Unwillingness of farmers to use fertilizers due to lack of awareness (Chelia) - Lack of capacity of the local contractors to complete shallow water well in time (Jima

Arjo) - Delay in road construction due to limited technical and financial capacity of the local

contractors (Shebedino, Dilla Zuria)

96. It is quite obvious from above that some of the problems/constraints of the regions/woredas are generic in nature and have been continuing for long viz. ‘shortage of skilled/trained manpower’, ‘high staff turnover’, and ‘budget constraints for operating expenditure and capital investment’ especially at woreda level. The programs like PSCAP, PBS, and ULGDP have partially addressed these problems

25


through capacity building and budget support. But these programs have not been able to address the issue of high staff turnover. Regarding the sector specific problems/constraints, the regional and local governments need to develop joint action plan to deal with them.

5.2.2 Quality of service delivery (FGD findings)

97. During field mission, FGDs were conducted with the community members in eight kebeles. Altogether 188 community members participated in the FGDs. The FGD participants were asked to respond, among others, to the following questions related to quality of service delivery viz. (i) whether they see any improvement in quality of basic services compared to previous two to three years, and (ii) what are the reasons for saying service quality has ‘improved’ or ‘not improved’. The findings are summarised in the following paragraphs45.

98. Primary education (n=170): A majority of the community participants (84.1 per cent) feel that quality of education has improved marginally, and smaller number of them (9.4 per cent) considers that primary education has improved significantly. The perception of improved service underscores several factors such as easy access to schools; improved infrastructure; availability of sufficient numbers of qualified teachers; education in mother tongue; and no discrimination in enrolment of students. However, some participants (6.5 per cent) in Libon Gamo Kebele (Cheliya woreda, Oromiya) do not see any improvement in quality of primary education in recent years.

99. Health (n=166): Like in the case of education, many community members (75.9 per cent) say that the quality of health services has improved marginally, while about a fourth of them (24.1 per cent) rate improvement in services to be significant. The reasons cited for their perception are as follows: improved access to health facilities; availability of adequate number of health extension workers; availability of ambulance services; reduced waiting time at the health facilities; and increased availability of family planning and contraceptive services.

100. Agriculture extension services (n=165): Most of the community members (78.8 per cent) feel that quality of agriculture extension services has improved marginally. A smaller proportion of the participants (10.9 per cent) consider that services have improved significantly. The community perception of improved services stem from a number of factors such as increased supply of improved seeds and fertilizers; support provided by development agents; and specialized training and advises. However, all the community members (10.3 per cent) in Chechu kebele (Dilla Zuria woreda, SNNPR) do not see any improvement in the quality of extension services.

101. Water supply (n=161): Regarding quality of water supply, the opinions of the community members are divided. Some of the community members (47.8 per cent) feel that quality of water supply has marginally improved, while some (10.6 per cent) rate the improvement significant. But, the community members (41.6 per cent) in Harores kebele (Jigjiga woreda, Somali), Tome kebele (Shinile woreda, Somali), and Chechu (Dilla Zuria woreda, ANNPR) do not see any improvement in the situation characterised by unclean water; non-functional water schemes; and difficulty in accessing drinking water sources.

102. Sanitation (n=167): Over half of the community members (52.1 per cent) find significant improvement in use of latrines, while some others (39.5 per cent) see only marginal improvement. There are number of reasons for improved use of latrines such as increased level of awareness; training on hygiene and sanitation; shame attached with open defecation; and improved availability of water. However, a small number of community participants (8.4 per cent) feel that access and use of latrine has not improved in recent years.

103. Rural road (n=165): A good number of community members (50.9 per cent) feel that quality of rural roads has improved marginally, while some of them (10.9 per cent) see significant improvement. The perception of improved rural road underscore good road conditions; connectivity with the main road, connectivity with the woreda town and local market; etc. However, the community participants

45 The number of valid responses (n) is shown in parenthesis.

26


(17.6 per cent) in Seiga-22 kebele (Asosa woreda, Benishangul-Gumuz) state that quality of rural roads has worsened, while the participants (20.6 per cent) in two kebeles viz. Dabus kebele (Bambasi woreda, Benishangul-Gumuz) and Chechu kebele (Dilla Zuria woreda, SNNPR) do not see any improvement.

104. On the whole, the FGD findings suggest that most of the community members at kebele level feel that basic services have improved marginally compared to three years before. The community perceive that education, health, agriculture, and sanitation services have been better than water and rural road services.

5.3 Progress in Strengthening Local Accountability and Transparency

5.4.1 Citizen’s engagement

105. Financial transparency and accountability: The FTA initiatives are well integrated with the government’s public financial management system, and impressive progress has been made. About 97 per cent of the woredas now post budget and expenditure using revised template, and in the process informing 29 per cent citizens. Among the service delivery units, 67 per cent disclose information using standardised service delivery templates. Multi-media are used to disseminate FTA information, such as notice/bill board, TV, radio, pamphlets, training, and pocket calendar. Initiatives are also underway to introduce procurement templates.

106. Alongside shifting to revised FTA templates, many regions have introduced result focussed FTA reporting format to collect information from woredas. Of the four regions visited by the evaluation teams, SNNPR and Somali regions are receiving woreda reports in the result focussed FTA format. The other two regions, Oromiya and Benishangul-Gumuz, are not using the new reporting formant46.

107. A FTA assessment was conducted in 2013, which inter alia reveals that 42.5 per cent of the households have seen the FTA templates posted in their jurisdictions47. The percentage varies across the regions. In Benishangul-Gumuz, 86 per cent of the households have seen posted templates, whereas proportion of such households is only 3 per cent in Somali. The study also shows that of the households who have seen budget information through posted template, 37 per cent have discussed it with their respective woreda/City officials, while 26 per cent have discussed among themselves.

108. In course of field mission, the present study has also come across some anecdotal evidence that suggest that citizens indeed act on FTA dissemination of budget and expenditure information. A few cases are as follows:

• Wocjema primary school (Dilla Zuria woreda, SNNPR) posted its annual plan and budget, but funds utilisation and progress was not satisfactory. The parent teacher association lodged a complaint with the woreda education office, which in turn engaged an auditor to check the accounts of the school. The auditor found misappropriation of funds by the school principal, and he was removed from the post of principal and retained just as a teacher. In addition, the misappropriated amount was recovered from his monthly salary.

• In Harawe kebele (Shinile woreda, Somali), the farmers training centre (FTC) was not functional since it was not located near village. The centre was not equipped with materials and lacked water facility. After the community came to know about agriculture budget, they took initiative to make FTC functional.

• Being aware of the budget disclosed through FTA board, the community in Shenile kebele (Shinile woreda, Somali) asked for reasons why the kebele did not have electricity, especially for irrigation works, despite budget allocation.

46 It was reported at April 2015 JRIS that Addis Ababa, Dire Dawa, Harari, Somali, Afar, Gambella and SNNPR regions received reports from the woredas as per the result focussed FTA format. 47 MoFED (Sept. 2013): Financial Transparency & Accountability Implementation Assessment Report

27


• The budget for rural road and bridge construction was posted to inform the community in Dila Aferara kebele (Shebedino woreda, SNNPR). As per construction budget, the contractor was supposed to use new wood. But the community noticed that the contractor was using old wood, and they lodged a complaint with the road office. The contractor was asked to use new wood as per budget provision.

109. One of the avowed objectives of FTA is to increase citizen’s understanding of the budget and expenditure posted through different media. All regions have been conducting Budget Literacy Training (BLT) for citizen, sector officials, council members, and the PFM team members. So far 420 thousand citizens have attended BLT48. However, female participation in BLT continues to be a challenge and needs special attention49. Apart from low female participation, there are other on-going challenges affecting FTA viz. frequent turnover of trained PFM members, and lack of commitment of some PFM team members50.

110. Social accountability: Under ESAP-2, the social accountability activities are implemented by 48 SAIPs in 223 woredas as against target of 150 woredas51. As per latest information, 39,872 citizens and 16,154 local officials and service providers have received training on how to use social accountability tools (community scorecard, citizen’s report card, participatory planning and budgeting, gender responsive budgeting, public expenditure tracking survey, etc.)52. There are 272 SAP wordas that have developed joint action plans for service quality improvement based on interface meetings between service users and providers53.

111. The results framework of PBS-3 has set a PDO target that the woreda officials would seek views of 50 per cent people in their kebeles on improving quality of basic services. This target has not been achieved during 2012-13 and 2013-14. According to ESAP-2 baseline survey (29 May to 6 June 2013), those who have been consulted on improving basic services in their kebeles range from 22.3 per cent for rural roads to 33.1 per cent for health services. The present evaluation also enquired in all the eight woredas visited about the progress made in consulting the community members at kebele level for improving basic services. None of the woredas could furnish any information in this regard.

112. It is claimed that ESAP-2 along with other interventions has facilitated service improvements in all the five basic service sectors. The improvements, among others, include addition of school classrooms; recruitment of additional teachers, qualified health personnel, DAs; construction of roads; and new water points (Table 5.3).

113. There are potential benefits of FTA-SA linkages, and these are now well understood by the government officials, service providers, SAIPs, and PFM teams54. In recent period, initiatives towards FTA-SA linkage have progressed a lot. MoFED has prepared and disseminated FTA-SA linkage guideline, action plan, and standard format of memorandum of understanding (MoU) to all BoFEDs. BoFEDs have already signed MoUs with SAIPs In Afar, Tigray, Harari, Somali, and Addis Ababa, and there is at least one member in the Social Accountability Committee (SAC) from PFM team55. The FTA team is also participating along with SAC in the interface meeting between community members and the service providers.

48 See PAD (August 19, 2015): ESPES, Para 22, p 51. 49 The low female participation is on account of cultural factors and household workload, which prevent them to leave home and attend BLT at woreda towns. There is a need to focus on increasing female participation, and one option could be to organise BLT at cluster of kebeles rather than at woreda towns. Refer to presentation (the case of FTA and GRM) at April 2015. JRIS. 50 Presentation (status of deliverables, FTA updates, and next actions) at April 2015 JRIS 51 ESAP-2: Quarterly Progress Report 14, April-June 2015 52 Updated Logical Framework, 31 December 2015 53 Ibid 54 For a detailed discussion on the subject refer to MA (Feb 2014): FTA-RSAP-2 Linkage Workshop 55 Presentation (status of deliverables, FTA updates, and next actions) at April 2015 JRIS

28


Table 5.3: Service improvements due to social accountability facilitations

Education Health • 153 additional student classrooms built, • 269 additional qualified teachers hired, • 3,585 student benches procured • 67 school toilets constructed, separately for boys

and girls

• 94 qualified health personnel were added • 53 water and sanitation facilities constructed/

maintained in health facilities • 14 wards and waiting room renovation and

construction • Pharmacies equipped with the necessary drugs

Agriculture Water & sanitation • 63 agriculture extension agents added • 24 additional veterinary staff recruited • 49 FTC and DA residences constructed • Community demand for fertilizers as well as inputs

increased, and government is supplying timely

• 99 water points were repaired and maintained, • 104 new water points built • 109 water points fenced • 762 community latrines built

Rural roads • 134 km of rural roads constructed (not covered by URRAP) • 247 km of rural roads were repaired and maintained. • 11 road experts were hire as rural roads officers • 49 community bridges constructed Source: Presentation at April 2015 JRIS

114. Grievance redress mechanism: The institutional structure of GRM in Ethiopia has evolved over years, and it consists of Ethiopian Institution of the Ombudsman and GRM offices at regions, zones, and woredas. As a Constitutional requirement, EIO was set up in 2000 through a Proclamation (No. 211/2000) with the objective of preventing and rectifying maladministration and promoting good governance56. At present, it has branch office in Tigray, Amhara, Oromia, SNNPR, Dire Dawa, and Gambella and there is a plan to open branch office in all the remaining regions. Legally EIO is independent and accountable only to the House of Peoples’ Representatives. But some concerns about its independence still persist because of dominance of the executive organs. Besides, the proclamation does not clarify how the institution can enforce its decisions. EIO is also facing the challenge of budget constraint and lack of capacity.

115. The GRM offices are present in all regions except in Afar. The kebele manager serves as a focal point for the GRM office at the kebele level. Although many of the regional GRMs have yet to clarify and define their GRM processes and procedures, there is more or less similar GRM legal framework in four regions and two city administrations. The remaining regions are yet to make progress in developing their respective legal framework and are mainly basing their GRM processes on the business process reengineering. The regional government funds GRM offices, and in turn they are required to report to the government. A recent study has brought out several challenges of the GRM system at regional/zone/woreda levels, such as lack of independence, and weak enforcement of decisions57. The same study also points out that the citizens are generally not satisfied with redress of complaints by the GRM offices. This is a matter of concern and deserves immediate attention.

116. PBS-3 has supported EIO by providing financial and technical support. The program also supported capacity building of the existing GRM offices in the region by developing a common standard of grievance redress procedures and handling, and by imparting training to the grievance handling officers. The PBS-3 targets of setting fully functional GRM offices in five regions in 2012-

56 Ethiopia is now one among the 100 countries that have set up ombudsman institutions. See World Bank Institute: The Role of Ombudsman Offices in Promoting Good Governance and Effective service Delivery (http://siteresources.worldbank.org/CSO/Resources/Summary_Roundtable_Role_of_Ombudsman_Good_Governance.pdf) 57 IPE Global: Studying the Grievance Hearing and Redress at Federal, regional and Local Levels in Ethiopia

29


13 and in six regions in 2013-14 have been fully achieved. Regarding preparation and adoption of standardised manual and guidelines for GRMs, target was not achieved in 2012-13. Five regions against a target of six regions prepared and adopted standardised manual and guidelines in 2012-13. But in 2013-14, the number of such regions increased to eight and the target was achieved.

5.4.2 Public financial management and procurement

117. The PFM system at the woreda level has matured over the years. One notable achievement is the roll out of IBEX in 800 woredas. Besides, a new financial management system called Integrated Financial Management Information System (IFMIS) has been under implementation in ten pilot budget units, including MoFED58. These units are expected to go live shortly with the new IFMIS.

118. In 2013-14, audit coverage by ORAG was 40 per cent and met the target. But, the coverage was less compared to 45 per cent coverage (against a target of 35 per cent) achieved in the previous year. Furthermore, only 5.1 per cent of the woredas acted on the prior period audit findings during 2013-14 as against the target of 25 per cent. The performance was better in the previous year, with 15 per cent woredas (against a target of 20 per cent) following up the audit findings.

119. The non-compliance with the audit findings by a large number of woredas is a serious concern, especially in view of the chronic weaknesses in woredas’ public financial management such as weak property management; long outstanding advances and payables; missing documents; delay in bank reconciliations; misclassification of expenditure; weak cash management; and weak internal control of payments59. The non-compliance with the audit findings is also a big set back, given the initiatives taken to ensure audit follow up such as appointment of audit committees in 400 woredas; establishment of an internal audit task team by MoFED to support the audit committees; various supervision missions undertaken by MoFED/BoFED; and repeated raising of this issue in the PFM technical working group (TWG) and DP-GoE meetings60.

120. Regarding external audit, the coverage at the federal level by OFAG has increased from 90 per cent in 2010-11 to 100 per cent in 2013-14. This is a remarkable achievement. The 2014 PEFA report further reveals improvement in timely response to audit observations by auditees.

121. Procurement is a significant challenge at woreda level. On average, a woreda procurement unit provides procurement service to 23 sector offices following a centralised procurement system (pool system)61. This system suffers from many shortcomings, including delay in procurement62. The woredas are also constrained by lack of technical skills to deal with works procurement and contracts management. The lack of technical skills may reflect shortage of full-time staff and/or lack of trained and experienced staff. There are also gaps in procurement planning, preparation of bid documents, bid evaluation, and contracts management63. It is imperative to equip woredas with professional/trained procurement staff and address the systemic bottlenecks in the procurement system.

122. For transparency and accountability to public, the woredas are required to post procurement award decisions. In this regard, progress has not been satisfactory. In 2013-14, 254 woredas posted procurement information as against the target of 400 woredas. It is understood that the procurement templates have been prepared and would be disseminated to the woredas for posting procurement information. The availability of template is expected to encourage posting procurement information.

58 PEFA Final Report, April 2015, p 64 59 PAD (August 19, 2015): ESPES 60 See DfID (2014): Annual Review, p 14-15 61 PAD (August 19, 2015): ESPES, p 121. 62 According to Aide Memoire, 5-16 Nov 2012 (p 8), an assessment of the pool system was carried out in 2012. The evaluation team could not get this report for a detailed study. 63 See for details PAD (August 19, 2015): ESPES

30


5.4.3 Managing for results

123. In terms of monitoring PBS results, MforR has performed reasonably well. The results matrix is updated and progress against higher-level and PDO indicators with regional disaggregated data (wherever available) are shared in every JRIS. Of course, the JRIS discussions on results tend to focus more on quantitative achievements and less on qualitative issues64. For instance, discussion on gender disparity in April 2015 JRIS mostly centred around on newly available data. There was hardly any deliberation on the factors underlying the gender inequities, and how to address those factors.

124. A core objective of MforR is to strengthen multi-sector MIS with data quality assurance. In this regard, the progress has not been uniform across the sectors. EMIS has been rolled out in 817 woredas as against the target of 600 woredas. In the health sector, the target for HMIS implementation in 90 woredas has been over achieved. However, there have been significant delays in MIS implementation in WASH, rural road, and agriculture sector due to delay in procurement of IT equipment at federal, regional, and district offices65. WASH-MIS has been implemented in 330 woredas, achieving only 55 per cent of the target, while none of the regions has rolled out MIS in road and agriculture sectors. Recently, some progress has been made. For instance, MIS software development, verification, and validation have been completed in the road sector66. Besides, 50 per cent of actual data collection on census of national network is completed67. In agriculture sector, development of Excel based MIS has been completed68.

125. Under MforR, data quality assurance has progressed quite well. Using Ethiopia Data Quality Assessment Framework (EDQAF), CSA has assessed data quality of education sector twice (2012, 2015), health sector once (2014), and road sector once (2012). Thus, CSA deliverables are met as per targets set in the results framework.

126. CSA rates data quality of education acceptable. But, there are problems related to timeliness and completeness of data reported from Somali, Benishangul, Harai, and Dire Dawa69. Besides, EMIS & IT Directorate currently has four experts for data management at national level70. The directorate urgently needs to recruit statisticians who can assist preparation of analytical reports that are currently missing. It also sees the need for having one statistician in each region. The other challenges inter alia include lack of infrastructure to exchange and linking data online, delay in data collection in some regions, and absence of regional level EMIS in most regions.

127. Regarding HMIS, CSA finds data management and storage is strong at regional level (except in Afar, Gambella, and Somali) but weak at lower levels (e.g. health post, health centres, and woreda). Data accuracy is a challenge. Addis Ababa, Benishangul Gumuz, Harari, Oromiya, and Tigray are found to be over-reporting and Afar Gambella under-reporting the number of 1st visit ANC. It may be further noted here that data quality assurance/control mechanism has not been developed to check data accuracy

128. The progress of MforR on analytical works has been slow. DPs therefore urged the government to review and priorities pending studies, and accordingly six analytical studies have been dropped71. The recently completed studies inter alia include the following studies: mini-EDHS, SPA+, gender disaggregated development data, education SDI survey, M&E assessment studies for education and health sectors, and DA tracking study. There are some studies that have been delayed considerably

64 This assessment is based on the experience of attending April 2015 JRIS. Some DPs also felt in the same way 65 DfID (2014): Annual Review, p 20 66 Presentation at April 2015 JRIS 67 Ibid 68 Ibid 69 CSA (March 2015): Overall Data Quality Assessment for Education Sector in Ethiopia 70 This and other information are gathered through interview with the Director, EMIS & IT Directorate. 71 For details about the studies dropped see PBS Secretariat (March 2015): PBS 3: B3 Managing for Results (MfR) Component - Status Update

31


such as cost of service delivery, benefit incidence study, and water sector data quality assessment. All these studies are important and should be expedited.

129. In view of the limited progress in some areas as discussed hitherto, the forward work plan of MforR has been redrawn and prioritized, and budget allocation has been reduced.


Program effectiveness (progress in achieving results)

Effectiveness of basic service delivery

Progress towards achieving higher-level and PDO targets: Of the seven higher-level indicators, targets were achieved only for three indictors (43 per cent) in 2012-13, and for four indicators (57 per cent) in 2013-14. Similarly, of the six PDO indicators, targets were achieved for three indicators (50 per cent) and four indicators (67 per cent) in 2012-13 and 2013-14 respectively. All the higher-level and PDO targets were missed in the water sector whereas all the targets were achieved in the road sector. Both these sectors require high capital investments, and their performance has little or no bearing with the PBS program.

Primary education: NER (Grade 5-8) and proportion of qualified teachers (Grade 1-8) remained below the targets in 2012-13 and 2013-14, while targets related to GPI and the pupil-teacher ratio are fully met in these two years. The lower primary education (Grade 1-4), which is not tracked in the results framework, is facing the problems of high pupil-teacher ratio and significant shortage of qualified teachers. The teachers’ motivation to take class was also found to be low.

Health: Penta-3 coverage targets were underachieved in 2012-13 and 2013-14, while ANC coverage exceeded targets. In 2013-14, norm for population per HEW (2,500 per HEW) was almost achieved. Some of the sector issues that are not tracked in the results framework are the following: low SBA deliveries; normal delivery service available in less than half of the health posts; and short supply of essential medicines in most of the health posts. Mixed progress in achieving health MDGs. While IMR has declined considerably and U5MR achieved ahead of time, MMR continues at a very high level.

Agriculture: The number of DAs with specialisation has not grown as per the targets set for 2012-13 and 2013-14. The problem of DA retention and quality of service continues unabated. However, food crop productivity targets nearly achieved, and outreach to direct beneficiaries (smallholder farmers and pastoralists) fully met the targets.

Water and sanitation: Rural water supply coverage and reduction in non-functional water schemes in 2012-13 and 2013-14 remained far below the targets. At present, about two-third of the households use some form of latrine, and one-fourth of the kebeles are open defection free.

Rural roads: Due to improved rural road connectivity, time to reach nearest all-weather road reduced and the targets set for 2012-13 and 2013-14 were almost achieved. Also, the targets of maintaining the roads in fair and good conditions were fully achieved. Besides, the road sector has established and maintained 500 woreda road desks with appropriate staff.

Mission findings on basic service delivery

Findings based on KIIs in select regions/woredas: Most of the woredas follow the practice of grading their primary/secondary schools using some criteria, while much lesser number of woredas evaluates the performance of the health facilities. Both the findings indicate opportunity for developing an appropriate ‘results based, and incentivised’ system to monitor service delivery at provider level.

The woredas are faced with some recurrent challenges/constraints such as shortage of skilled/trained manpower, high staff turnover, and budget constraint for operating expenditure and capital investment. Besides, there are some sector problems (see text for examples).

Findings based on FGDs: Most of the community members at kebele level feel that basic services have improved marginally compared to earlier years. They also perceive that education, health, agriculture, and sanitation services have improved relative to water and rural road services.

Progress in strengthening local accountability and transparency

Citizen engagement: The FTA initiatives have been well integrated with the GoE’s public financial management

32


Program effectiveness (progress in achieving results) system, and impressive progress has been made. All the FTA targets were achieved in 2012-13 and 2013-14. The field mission also gathered anecdotal evidence to suggest that citizens acted on FTA dissemination of budget and expenditure information.

Social accountability: The social accountability activities have progressed significantly, surpassing the targets set for 2012-13 and 2013-14. The SA facilitation together with other interventions has also brought about commendable improvements in the basic service delivery. However, community consultation by woreda officials for improving basic services has not progressed as expected. In terms of FTA-SA linkage activities, there has been satisfactory progress in the recent period.

Grievance redress mechanisms: EIO has branch office in six regions, and GRMs are in place and functional in almost all the regions. Besides, all regions have adopted standard manuals and guidelines. A recent study has brought out several challenges of the GRM system at regional/zone/woreda levels, such as lack of independence, and weak enforcement of decisions. The same study also points out that the citizens are generally not satisfied with redress of complaints by the GRM offices.

Public financial management and procurement: The PFM system at the woreda level has matured over the years, and the progress includes: rolling of IBEX all over the country except in few woredas; improved audit coverage by ORAG; and complete audit coverage by OFAG. However, a large number of woredas do not follow up actions on audit findings.

The woredas are constrained by lack of technical skills to deal with works procurement and contracts management. For transparency and accountability, the woredas are required to post procurement award decisions. Not much progress has been made in this regard.

Managing for results: The updating and sharing of PBS results in every JRIS has progressed well, but the work related to strengthening multi-sector MIS and conducting analytical studies has been slow. Some areas of concern are: slow roll out of WASH MIS; failure to roll out MIS for agriculture and road sectors; and delay in conducting some critical studies (e.g. cost of service delivery, benefit incidence study, water sector DQA). In view of such slow progress, the forward work plan has been redrawn and prioritized, and budget allocation has been reduced.

Chapter 6: Aid Effectiveness of the PBS Program

The present evaluation has looked into aid effectiveness of the PBS program in the background of the aid landscape in Ethiopia. The assessment is carried out in this chapter using the Paris Declaration. This chapter also includes a brief discussion on the PBS-donors’ engagement in policy dialogue with the government. Currently, donors’ harmonization has reached a cross road, given the realignment of their support to basic services. The donors are now developing a new framework for harmonization. The initiatives taken in this regard are briefly noted in this chapter.

6.1 Aid Landscape in Ethiopia

130. A large number of active development partners are operating in Ethiopia72. In 2013 (the latest year for which OECD-DAC data are available), total official development assistance (ODA) was USD 3.9 billion73, of which ‘aid on budget’ amounted to USD 2.9 billion (74 per cent)74. Over the years, ODA flows to the country have shown two discernable trends. First, humanitarian aid has declined, reflecting the country’s improved resilience to cope with humanitarian emergencies75. Second, share of concessional loans in ODA portfolio has increased compared to grants indicating Ethiopia’s growing debt sustainability capacity76. It may be further noted that despite growth in ODA volume, the country’s aid per capita continues to be lower than the average of Sub-Saharan Africa77. 131. Since its inception in 2001, the development assistance group (DAG) in Ethiopia has been engaged with the government to promote effective delivery of ODA, and to foster cooperation and coordination among all the development partners and the government. DAG now comprises of 27 bilateral and multilateral DPs, and its overarching aid coordination structure consists of (i) DAG’s own set up (executive committee, secretariat, and technical working groups), (ii) MoFED’s Effective Development Cooperation Task Force (donor-government sector working groups), and (iii) donor working groups78. DAG holds regular dialogue with the government through the annual GTP progress review meeting and the high-level forum79. In addition, MoFED convenes the Effective Development Cooperation Task Force to further promote adherence to the development effectiveness agenda in the country. The task force meets every quarter and follows up on the implementation of aid effectiveness plan and targets of the country. The task force also carries out groundwork for policy dialogue between the government and DAG in high-level forum. 132. In 2007, aid management platform (AMP) was launched to enhance transparency, mutual accountability, and efficiency in development cooperation. AMP refers to a web-based aid database, which is managed by MoFED. Currently more than 24 donors have AMP workspace, which can be directly accessed by them to enter their commitment and disbursement data80. AMP guidelines clearly state the role and responsibility of donors and MoFED. However, the donors have not yet started entering the data81. In 2013, MoFED published its annual ODA Statistical Bulletin presenting on-budget aid using EFY 2005 AMP data. The bulletin was distributed to all development partners.

72 According to the aid management platform data, there are 22 bilateral and 24 multilateral DPs. See DAG Annual Report 2013-14. Profiles of 41 DPs are available in another DAG publication (July 2015) titled Profiles of 41 Development Partners in Ethiopia. 73 DAG (July 2015): Profiles of 41 Development Partners in Ethiopia, p 4 74 Annual Statistical Bulletin EFY 2006 (AMP data) 75 For instance, Ethiopia fared well during 2011 Horn of Africa drought compared to the neighbours. 76 DAG (July 2015): Profiles of 41 Development Partners in Ethiopia, p 4-5 77 In 2012, per capita aid in Ethiopia was USD 36 compared to Sub-Saharan average of USD 50 per capita. See http://www.dagethiopia.org. 78 For further details refer to DAG Annual Report 2013-14 79 MoFED chairs the high level forum, and heads of delegation (i.e. ambassador level) meet with the ministries in this forum regularly to discuss key development cooperation issues. 80 Information gathered from MoFED 81 Ibid

34

Process Evaluation of PBS-3, Final Report: Aid Effectiveness of the PBS Program

Going forward, the DPs and the government are required to institutionalize the use of AMP data to inform future program decisions. 133. In 2013, post-Busan monitoring survey was conducted in Ethiopia, contextualizing it with the aggregate results for 46 countries82. In total 24 DPs in Ethiopia participated in monitoring exercise, which captured 82 per cent of the total ODA in the country during July 2011-June 2012. The monitoring results show that Ethiopia scored above the global average for select indicators of aid effectiveness (viz. aid predictability, aid on budget, mutual accountability, gender equality, and use of the country system) albeit there was a decrease in score on the use of national PFM and procurement systems compared to the last evaluation in 201083. Based on the results, an action plan was put in place to make further progress84. 134. In terms of aid effectiveness at program level, it is pertinent to note that the PBS and PSNP programs were featured as showcases on aid effectiveness at the fourth high-level forum held in December 2011 in Busan, South Korea85.

6.2 Aid Effectiveness of the PBS program

135. The present evaluation has assessed the aid effectiveness of the PBS program in terms of the five principles of the Paris Declaration of aid effectiveness viz. ownership, alignment, harmonization, managing for results, and mutual accountability. The findings suggest that the PBS program, as evolved through phases, meets all the criteria demonstrating a high degree of aid effectiveness.

136. Ownership: The GoE engagement in the design of PBS-1 was limited, but involvement matured in the successive phases leading to complete ownership of the program design including the core principles and the results framework. The government ownership and deep commitment is also evinced by the fact that its share in block grant transfers increased from 58.1 per cent in 2006-07 to 85.5 per cent in 2014-15.

137. Alignment: The development objectives of the PBS program are well aligned with four out of seven pillars of GTP (see Chap.4, Sec. 4.2.1). Besides, the program makes use of the government’s Treasury systems and accounts to transfer block grants to the woredas. For performing tests related to core principles, as well as for monitoring results, the program mainly relies on GoE’s administrative database and M&E system.

138. Harmonization: The pool-funding system of the PBS program through multi-donor trust funds (managed by the World Bank) has been a key instrument to achieve donors’ harmonization. The pool-funding system has not only reduced the transaction costs, but also relieved MoFED from following multiple financial procedures and reporting systems that the individual donors might have imposed86. The other key mechanisms that have supported donors’ harmonization are following:

• JRIS missions: The semi-annual JRIS mission provides a single platform to all donors, the federal and regional governments, management agency, and other government and non-government agencies for developing a shared understanding of the results, progress, and challenges of the PBS program. This common platform has eliminated taxing the government with multiple donor missions.

82 MoFED conducted the exercise under the guidance of the Effective Development Cooperation Task Force. The DAG secretariat facilitated cooperation between DPs and GoE. 83 For detailed monitoring results see DAG (June 2014): Post-Busan monitoring results for Ethiopia 84 Although Ethiopia does not have an aid policy or partnership policy, the DAG and MoFED have previously agreed on an aid effectiveness action plan with clear indicators to measure. See DAG (June 2014): Post-Busan monitoring results for Ethiopia, p3 85 See www.africa-platform.org (search Ethiopia) 86 In the absence of data, the present evaluation could not estimate savings in transaction costs, net of the Bank’s funds management fees and the PBS secretariat’s operating costs.

35


• PBS donors group: The PBS donors meet monthly to discuss the various issues related to the program implementation and to take joint decisions. The World Bank holds the permanent chair of this forum, while the co-chair rotates among other donors. A close scrutiny of a sample of minutes of the meetings shows that the system of monthly meeting has been quite effective in fostering collective decision-making and DP harmonization.

• PBS secretariat: The decision to use dedicated resources to set up a PBS secretariat has been a critical design feature of the PBS program. Both the donors group and MoFED appreciate the wide-ranging activities that the secretariat undertakes to facilitate harmonization and coordination among DPs, and between the government and DPs.

139. The pool funding system and the other mechanisms as described above together present a well-harmonised donor position to the government, which in turn makes the government’s task of donors’ engagement smooth and manageable. In view of this, it is quite reasonable to say that the PBS program fully meets the harmonization principle of aid effectiveness.

140. Managing for development results: The PBS-3 program has a stronger focus on results. In particular, the program design includes a specific component viz. managing for results. The program results are updated and discussed in every JRIS, and the measures to improve the results and to address specific challenges are jointly decided by GoE and DPs.

141. Mutual accountability: The PBS program ensures high level of mutual accountability through semi-annual JRIS/JBAR (which also includes pre-JRIS field visits to select regions/woredas). The key aspects of JRIS/JBAR that pave mutual accountability are the following:

• Joint review of results/deliverables: JRIS/JBAR is the primary mechanism by which the government and DPs undertake a joint review of (a) program results, progress, and challenges; (b) application of the core PBS principles; and (c) financial management of the program. The review process promotes GoE-DP dialogue, leading to joint agreement about the action plan going forward.

• Trigger for disbursement of funds: The satisfactory outcomes of JRIS/JBAR trigger release of block grant (under Sub-Program-A)87.

• Deepening accountability: JRIS/JBAR is attended by a large number of sector and regional representatives. This deepens mutual accountability for results beyond MoFED-DP level, and involves stakeholders from the regions, sectors, government agencies, and CSOs. However, in recent years, participation of line Ministers and other high-level government officials in JRIS has been uneven, causing a bit of concern 88.

142. There are other initiatives that have deepened mutual accountability such as participation of channel one programs coordination office (COPCO) of MoFED in alternative monthly meeting of DPs, and the MoFED directors holding the chair of technical working groups that are lagging behind and demanding greater participation from the sector ministries.

6.3 Donors’ Engagement in Policy Dialogue with the Government

143. Historically, budget support system provided opportunity to the donors’ for engagement in policy dialogue with GoE on macro issues such as economic stability, poverty reduction, governance, human rights, and political accountability. This opportunity was embedded in the budget support instrument.

144. Since inception of the PBS program, JRIS/JBAR has become the main forum for GoE-DPs dialogue, which mainly centres on reviewing program results and developing operational plans to deal

87 See the loan/credit conditions and covenant section of PAD (29 August, 2012): Promoting Basic Services Phase III, p 27. 88 This information is gathered through interaction with the Bank staff.

36


with key implementation issues. The government has been very responsive to this kind of dialogue because in its view policy dialogue under PBS must focus on how to improve delivery, quality, and governance of basic services89. Going forward, the donors also agreed that macroeconomic dialogue should not be within the purview of semi-annual JRIS/JBAR, and following this understanding GoE and DPs have been continuing with technical exchanges on macroeconomic policy issues at quarterly meetings. The PBS secretariat has facilitated such technical exchange.

145. At the outset of PBS-3, GoE committed to set up an alternative forum, outside the main program, for having sustained dialogue on macroeconomic policy issues90. GoE took some time to set up this forum. In 2014, a macro dialogue forum was established. From the government side, MoFED and the National Bank of Ethiopia participate in this forum. With the setting up of this forum, there is now a formal mechanism for donors to engage in dialogue with GoE on macroeconomic policy issues albeit DAG continues to provide a bigger forum for any dialogue on economic/financial/sector issues.

146. The evaluation team further noticed that JRIS/JABR does not provide scope for strategic discussion and decision-making. Rather JRIS acts as a forum for sharing results, progress, and challenges with larger number of stakeholders; building a consensus regarding action plans to address the challenges; and agreeing deliverables for the next session of JRIS. This raises the question whether the format of six-monthly JRIS/JABR needs to be changed, or additional forum should be set up (e.g. high-level project steering committee) to create space for any strategic discussion regarding the PBS program, including discussion on resource mobilisation and institutionalisation of good governance practices to ensure expansion and sustainability of quality basic service delivery.

6.4 Recent Initiatives for Donors’ Harmonization

147. As mentioned in the introductory chapter (see Para 5, Chap. 1), the PBS donors have recently realigned their support to basic services, creating a new challenge for harmonization. Despite their different modalities to support the government, the donors are keen to have a common platform to sustain dialogue and harmonized efforts to promote basic service delivery.

148. The DPs initiated the development of a roadmap for harmonization in early 2015, and identified a few priority areas, such as developing knowledge management system; creating a new coordination entity replacing the existing PBS secretariat; revising JRIS process and format; setting up a knowledge forum; strengthening DAG engagement and recognition; building communication strategy; and developing sector engagement plans91.

149. As a follow up of the above roadmap, the PBS secretariat was reconstituted as SDS with effect from July 2015. After many deliberations, the DPs developed a ToR for SDS. It contains twelve overlapping objectives and seven functions92. The evaluation team finds this ToR less compact and focussed, and therefore requires revision.

150. Very recently (January 2016), DPs held a one-day workshop and further deliberated on the issues and various options of harmonization93. The workshop identified several weaknesses in the existing harmonisation system, such as lack of a common work plan; different structures and performance of the existing TWGs; and less scope for strategic dialogue and priority setting through JRIS/JBAR. The workshop also reached a number of broader agreements including, among others, the following: developing a common planning framework to reduce duplication of interventions, and identify areas of synergy; reviewing and updating the tools of harmonization including JRIS and TWGs; and increasing the engagement with regions and woreda

89 World Bank, IEG Group (Dec 2013): Project Performance Assessment Report (PBS), Para 6.9 90 Aide Memoire, Nov 5-16, 2012, p 4-5 91 PBS Secretariat (2015): Supporting a new phase of decentralized basic service delivery – roadmap paper for a harmonized approach 92 The evaluation team collected the SDS-ToR from the PBS secretariat. 93 For details see Draft Report (Version 2, 28 Jan 2016): Services Delivery Harmonization Workshop

37


151. The development of a roadmap for harmonization is still a work in-progress. The donors need to flesh out the action plans, and enter into a mutually agreed memorandum of understanding (MoU). Also, the SDS needs to be adequately staffed, and its ToR must be defined appropriately.


Aid effectiveness of the PBS program

Aid landscape in Ethiopia: Ethiopia scores above global average in terms of aid predictability, aid on budget, mutual accountability, gender equality, and use of the country system. Among the multi-donor funded programs, PBS and PSNP were featured as showcases on aid effectiveness at the fourth high-level forum held in December 2011 in Busan, South Korea.

Aid effectiveness of the PBS program: The present evaluation finds that the PBS program meets very well the aid effectiveness principles of the Paris Declaration.

Donors’ engagement in policy dialogue: Since inception of the PBS program, JRIS/JBAR has been the main forum for GoE-DPs dialogue regarding program results, and developing operational plans to deal with key implementation issues. Until the formation of a macro dialogue forum in 2014, GoE and DPs discussed macroeconomic issues at quarterly meetings. With the setting up of this forum, PBS donors now have an appropriate mechanism to engage in dialogue with GoE on policy issues. The present evaluation has also noticed that JRIS/JBAR does not provide any scope for strategic discussion and decision-making. This raises the question whether the format of JRIS/JABR needs to be changed, or additional forum should be set up (e.g. high-level project steering committee) to create space for any strategic discussion regarding the PBS program.

Recent initiatives for donors’ harmonization: The donors’ harmonization has now reached a cross road, given the realignment of their support to basic services. In this regard, the PBS secretariat has been reconstituted as a Service Delivery Secretariat with effect from July 2015. The donors are keen to develop a common framework for harmonization, and several rounds of discussion have already been taken place. The work is still in-progress, and the donors need to flesh out concrete action plans and enter into a memorandum of understanding. The SDS needs to be adequately staffed, and its ToR must be defined appropriately.

Chapter 7: Program Efficiency

This chapter deals with the evaluation of program efficiency, which includes assessment of (a) value-for-money (VfM), (b) program implementation, and (c) risk management. The VfM analysis has taken into account four criteria, namely economy, efficiency, effectiveness, and equity. The evaluation has mainly relied on review of secondary literature and analysis of secondary data.

7.1 Assessment of Value-for-Money

152. Cost drivers at woreda level: Analysis of woreda level expenditure in recent years (2011-12 to 2013-14) reveals that the education sector has 35.5 per cent share in total expenditure, while health and agriculture accounts for 11.1 per cent and 10.9 per cent respectively (Figure 7.1). Water and road sectors absorb very small shares in the total expenditure, 2.5 per cent and 1.4 per cent respectively. Thus, the combined share of the five basic service sectors in the total expenditure at woreda level is about 61 per cent.

Figure 7.1: Sector shares in expenditure at woreda level (2011-12 to 2013-14)

153. Salary payment to staff has remained the major cost driver of basic service delivery at woreda level, accounting for 78 per cent of the total expenditure (Figure 7.2). Among the sectors, share of salary in total recurrent and capital expenditure is highest in the education sector (90 per cent), followed by health (69 per cent), and agriculture (65 per cent). Water and road being more capital-intensive sectors, salary represents relatively smaller shares (42 per cent and 19 per cent respectively).

Figure 7.2: Cost structure of basic service (2011-12 to 2013-14)

9069 65

4219

78

624

18

16

8

12

4 717

4373

10

Educa1on Health Agriculture Water Road Allservices

(Percent)Source:ComputedusingMoFEDdata

Salary Opera1ngexp Capital

Road1.4% Water2.5%

Agriculture10.9%

Health11.1%

Educa1on35.5%

Others38.5%

39

Process Evaluation of PBS-3, Final Report: Program Efficiency

154. Economy: Given the predominance of salary component at woreda level expenditure (especially in education, health, and agriculture sectors), the economy aspect of VfM needs to be analyzed with reference to salary cost. In this regard, a DfID funded VfM study (hereinafter referred to as VfM study) reveals the following94:

• In Ethiopia, the pay scale of teachers and health professionals are better than other civil servants with similar qualifications95. However, compared to private sector, the experienced teachers and health professionals get lower pay. Furthermore, the salary of primary teachers relative to per capita GDP is significantly low in Ethiopia as compared to Tanzania and Uganda but slightly higher than Kenya.

• Index of real average unit costs of teachers in primary and secondary schools declined from 100 in 2010-11 (base year) to 76.16 in 2011-12, and then marginally improved to 78.3 in 2012-13. The index improved further to 82.10 in 2013-14, but still remained about 18 per cent lower than the 2010-11 level96. On the whole, teacher’s salary has declined in real terms over the period.

• All procurements related to Sub-Program-B have taken place as per the government systems which albeit suffers from several weaknesses. Recognising high procurement risks, PAD of PBS-3 proposed several risk mitigation measures. Presumably, risk oversight would ensure economy of procurement.

• The costs of program implementation support, largely staff related, are low. One contributing factor is small staff size of the PBS secretariat. The Bank’s fund management fee also compares well with the amount charged by other international agencies.

155. Based on the above findings, the VfM study concludes that block grant financing for basic service salaries under Sub-Program-A has achieved good value-for-money at economy-level.

156. The present evaluation, however, holds a different viewpoint regarding economy of the staff costs. Any decline in real staff costs, or prevalence of low pay structure in the public sector compared to private sector and other African countries, does not necessarily mean economy of staff costs. In the Ethiopian context, low salary structure is one of the key causal factors for staff turnover in the public sector. For a more meaningful analysis, the VfM study should have taken into account costs of staff turnover, including the benefits forgone of the training and other capacity building expenses incurred for those who left the basic service sectors. In sum, the present evaluation is of the opinion that low and declining real salary indicates ‘hidden problems’ and not ‘value’ for money.

157. Efficiency: Like the VfM study, the present evaluation finds some concerns about the efficiency with which the PBS inputs have been transformed into outputs. These are listed below97:

• Though absolute number of teachers has increased over the years, lack of qualified teachers at Grade1-4 still continues to be a challenge98.

• Despite the availability of health posts and health centres (nearly as per norms), as well as presence of a strong force of HEWs, the coverage of critical services like ANC (during 1st visit) declined, from 2.86 million in 2012-13 to 2.52 million in 2013-14. Similarly, the coverage of Penta-3 vaccine remained below the targets set for 2012-13 and 2013-14.

• The number of DAs with specialization of agro-ecological zones and high value crops has not increased as per targets set for 2012-13 and 2013-14.

94 ITAD (Feb. 2015): Value for money assessment of the promotion of basic services (PBS) programme 95 In Ethiopia, a pay revision took place at the start of 2014-15, after frozen pay scale since 2010-11. 96 The total salaries in basic service sectors at woreda level increased between 2006-07 and 2013-14 at a nominal growth rate of 23.8 per cent and at 1.7 per cent in real terms. The trend data are not shown here. 97 These points are drawn from Chap. 5. 98 The number of teachers increased from 171,079 in 2004-05 to 335,109 in 2012-13. See PAD, ESPES, p 15.

40


• Although most of the key output indicators covering Sub-Program-B were on track during 2012-13 and 2013-14, the progress under MforR was limited, with slow funds utilization.

158. It is evident from above that the PBS-3 program has not achieved the planned level of efficiency. There is a need for close monitoring of the results to improve efficiency.

159. Effectiveness: The VfM study suggests that the PBS has been a very effective program. The conclusion is based on (a) the findings of a recent poverty and social impact assessment (PSIA) study conducted by the Bank; (b) the DfID’s business case for PBS-3, and (c) draft PEFA report99. The VfM study has not done any independent analysis.

160. The DfID’s business case provides an estimate of benefit-cost ratio (BCR) of the PBS program. The estimated BCR is 1.43, which shows that benefits outweigh the costs. The BCR calculation is based on a set of assumptions, which the VfM study could check to a large extent for validation with the latest available data and information. The sensitivity analysis further reassures that BCR would be greater than one under different adverse scenarios.

161. In this context, the present evaluation is of the opinion that the VfM study could have undertaken independent cost benefit analysis instead of relying solely on DfID’s business case. For instance, the study could have estimated rate of return to schooling using Minecerian earnings functions. Such estimation has been carried out for GEQIP-II, which shows that rate of return to schooling is approximately 17 per cent in Ethiopia100.

162. Similarly, VfM study could have undertaken a cost effectiveness analysis for health sector by estimating cost of disability-adjusted life year (DALY) saved due to reduction in maternal and child deaths. Since the PBS program has supported basic mother and child services by providing additional HEWs, the DALY analysis could have provided some insights into effectiveness of the PBS program.

163. Equity: The PBS-3 and its predecessor programs have not made any visible impact in reducing inequities in basic services. The inequities in service delivery across the regions and income/wealth quintiles have persisted over the years (see Annex 4 for evidence of inequities in education and health sectors).

164. The VfM study further highlights the formula-based allocation of block grants from the federal to the regions, and from the regions to the woredas as GoE’s approach to foster equitable development in the country. It may be noted here that the regions apply varied allocation formulae, and many of them have not updated their formulae for long period of time101. Thus, there is a need to assess first the efficacy of the regional formulae in achieving equitable allocation of block grant to woredas.

7.2 Efficacy of the Program Implementation

165. Financial management of the program: As per the restructuring paper (20 July 2015), the revised budget estimate of PBS-3 is USD 1215.74 million (IDA fund of USD 600 million plus Trust Fund of USD 615.74 million). The program budget has shrunk by USD 162.4 million due to DfID’s withdrawal from the program102. Of the revised budget amount, USD 924.01 million (IDA fund of USD 489.5 million plus Trust Fund of USD 434.51) million was disbursed as of 19 May 2015. This represents 76 per cent disbursement of donors’ funding support to PBS-3.

99 See Qaiser M Khan et al (World Bank, 2014): Improving Basic Services for the Bottom Forty Percent; DfID (7 Nov. 2012): Business Case for Promotion of Basic Services in Ethiopia, 2012 – 2017; and Bank (April 2015): Public Finance Management Assessments: Integrated Report Based on Public Expenditure Financial Accountability (PEFA) Framework. 100 See the World Bank (Oct 17, 2013): General Education Quality Improvement Project II, Annex 6, Table 8. 101 For a good summary of approaches followed by different regions and cities, refer to the World Bank (March 2014): Integrated Fiduciary Systems Assessment – Ethiopia ULGDP II Program for Results, p 16-17. 102 World Bank (20 July 2015): Restructuring Paper, p7

41


166. Budget utilisation of Sub-Program B has not been as satisfactory as expected, especially the budget absorption under the sub-component MforR. As per PBS secretariat update, of the total budget of $22.7 million (revised budget) for MforR, around $13.8 million was spent or committed as of March 2015, representing funds utilisation of 60.8 per cent. Due to slow progress in funds utilisation, MforR activities have been revised and re-prioritised.

167. As on 30 June 2015, the management agency disbursed a total amount of USD 12.79 million to all SAIPs, which accounted for 89.5 per cent of the allocated grant amount of USD 14.29 million. Of the disbursed amount, actual expenditure was USD 10.72 million representing 83.8 per cent funds utilization103. In fact, funds utilisation under ESAP-2 has been better than other components under Sub-Program-B.

168. The quarterly IFRs, annual audit report, and concurrent audit reports are all submitted in time. However, there are some challenges. The IFR reports lack narration or explanation for the reported budget variance104. Also, there has been delay in submission of audit feedback and follow up actions are not taken to avoid repeated audit findings105.

169. The PBS audit report for 2012-13 was submitted in time and it did not contain any qualifications. But the management letter noted some findings related to lack of banks reconciliations, and some wrong posting of expenditures106. These issues have since been addressed with the regions refusing to release PBS funds without completion of bank reconciliations, and by circulating guidance on expenditure classification107. The audit report for the year 2013-14 was due in January 2015, but the present evaluation is not aware about the status of this report.

170. Bank’s performance: The bank’s overall performance in designing and implementation of PBS-3 has been satisfactory. The Bank put together a relevant and coherent PBS-3 program, building on six years of prior experience of program implementation as well as after giving due consideration to political, social, and economic developments of the country at appraisal stage. However, as discussed earlier, the Bank could have done better by addressing the issues of regional and income/wealth inequities in accessing the basic services, and by improvising the results framework to address the shortcomings as pointed out in this evaluation (see Sec 4.2.7, Chap. 4).

171. Turning to managing funds through MDTF, the Bank has performed well as acknowledged by key development partners like DfID. According to DfID, the Bank’s management of funds is satisfactory with the timely provision of quarterly IFRs and external audit report108.

172. The Bank’ overall leadership and program implementation support is laudable. It has ensured country-based presence of a qualified team to provide effective implementation support. With the support of PBS secretariat, the Bank has successfully conducted semi-annual JRIS/JBAR as per plans. As mentioned earlier, the PBS program meets very well the aid effectiveness principle. The credit goes to the Bank for its effective leadership, and the PBS secretariat for its support for donors’ harmonization and coordination. MoFED and other donors highly appreciate the Bank’s leadership and implementation support.

173. Government’s performance: Under the leadership of MoFED, the government has been highly supportive to PBS program implementation. The GoE‘s ownership and deep commitment is also evinced by the fact that it is the largest financial contributor to the PBS program. GoE has also shown deep respect for core principles, and wholeheartedly participated in all JRIS/JBAR. As mentioned in successive Aide Memoire, each year GoE has met the criteria of additionality and fairness tests.

103 ESAP Flash, Oct. 2015 104 Presentation on financial management at April 2015 JRIS 105 Ibid 106 DfID: Annual Review, 2014, p 23 107 Ibid 108 DfID: Annual Review 2014, p 23

42


Besides, over the six years from 2008-09 to 2013-14, the federal government fully transferred the budget amount of block grant to the regions109.

174. Performance of the PBS Secretariat: The PBS secretariat undertakes a wide range of activities, and it has played a crucial role to support program implementation, and to achieve donors’ coordination and harmonization110. Both DPs and MoFED appreciate the efforts put in by the secretariat. However, there are areas where the secretariat could have paid attention to add more value to the program.

175. For instance, effectiveness, sustainability, and equity reviews have relied on thematic, pre-JRIS field mission to a few woredas. The field mission often lacked specific focus on these principles, and findings based on few woredas are not good enough to draw any inferences. This is an area where the in-house expertise of the secretariat, the Bank, and other DPs could have been pulled together to produce specific studies to stimulate DP-GoE dialogue on the subject. The results framework is another area where the secretariat could have added more value by addressing the shortcomings as pointed out in this evaluation (see Sec. 4.2.7, Chap. 4).

176. In addition to above, the secretariat has not taken any initiatives to develop and implement communication and knowledge management strategy. Except publishing a few issues of PBS update during 2015, the secretariat has not produced any knowledge products (e.g. documentation of best practices). It is also noticeable that the secretariat has not hosted a website to post documents, reports, results and progress data. Such dissemination would have attracted comments, observations, and suggestions from various stakeholders including the beneficiaries.

7.3 Risk Management

177. At the appraisal stage, the Bank was aware of the risk of possible human abuse under CDP in Bennishangul-Gumz, Gambella, Somali, and Afar (see Annex 6, PAD 29 August 2012, p 97). However, no specific measures were considered to mitigate the CDP related risks. In this regard, the Investigation Panel observed that the Bank’s approach did not meet the standards of a systematic or holistic assessment of risks as set in the operational risk assessment framework (ORAF). The Panel also found this omission as non-compliance with the Bank’s OMS 2.20 on project appraisal. However, the Bank responded to the concerns raised by the panel with a comprehensive action plan111. In particular, the Bank has developed environmental and social screening tool to manage potential risks associated with any operational interface between Bank-financed projects and CDP. The government has already approved the tool. Besides, a safeguards implementation manual and training program are also under development, which will enhance the capacity of woreda for environmental and social risks management.

178. The PBS includes several other risk mitigation measures, such as (i) making provision for GoE-PBS dialogue on inflation and broader macroeconomic issues outside the purview of the PBS-3 program; (ii) JRIS review of progress towards developing sustainability through increased generation of domestic revenue; (iii) inclusion of FTA, SA, and GRM components to promote local governance

109 PAD (August 19, 2015): ESPES, p 47 110 For instance, during 2014, the secretariat carried out inter alia the following activities: (a) providing administrative, logistic and technical support for holding pre-JRIS field mission and JRIS; (b) coordinating involvement of senior representatives of line ministries/agencies in the meetings of the technical working groups, and providing DPs with regular updates of TWGs; (c) facilitating technical discussions on macroeconomic issued in a separate forum outside the PBS program; and (d) drafting the JRIS Aide Memoire and supporting DPs and the government counterparts to finalize the Aide Memoire so as to make timely disbursement to FBG. The team-lead of the present study also participated in April 2015 JRIS and witnessed the huge contributions of the secretariat in successful conduct of the JRIS. See Annual Report, Jan-Dec 2014; and Semi-Annual Report, Jan-June 2015. 111 See for details World Bank (31 January 2015): Management Report and Recommendation in Response to the inspection Panel Investigation Report.

43


and accountability, as well as GoE’s written assurance that the civil society law would not stop implementation of social accountability activities; (iv) setting up donor-funded PBS secretariat to provide program implementation support and aid effectiveness; and (v) several initiatives by MoFED to address recurrent challenges of financial management, including restructuring of COPCO. All these risk mitigation measures have been implemented.


Program efficiency

Value-for-money: The views of the present evaluation on VfM are the following. The declining real salary costs in the basic service sectors does not indicate ‘economy’ of staff input costs. PBS-3 has not achieved the planned level of ‘efficiency’. The ‘effectiveness’ of the program has been satisfactory, but equity in service delivery has not been achieved.

Efficacy of the program implementation: The overall funds disbursement was 76 per cent as of 19 May 2015. But, disbursement and utilisation of funds under Sub-Program-B excluding ESAP-2 has not been satisfactory. Funds utilisation under ESAP-2 has been better than other components of Sub-Program B. IFRs and audit reports are submitted in time, the latest external audit contains no qualifications.

The Bank’s leadership and implementation support has been satisfactory. GoE has shown sustained ownership and commitment for the program.

The PBS secretariat has played an appreciable role to support program implementation and to achieve donors’ harmonization and coordination. Of course, there are areas where it could have added more value, such as: producing specific studies on program effectiveness, sustainability, and equity so as to stimulate GoE-DP dialogue; improvising results framework to address the shortcomings highlighted in this study; and developing and implementing strategy for communication and knowledge management.

Risk management: The Investigation Panel observed that the Bank’s approach did not meet the standards of a systematic or holistic assessment of risks as set in the ORAF. However, the Bank has taken initiatives, among others, to develop environmental and social screening tool to manage potential risks associated with any operational interface between Bank-financed projects and CDP.

Chapter 8: Impact and Sustainability

This chapter is devoted to assess the PBS program using the criteria of impact and sustainability. Given that this is a process evaluation, impact assessment has not been carried out in details. The evaluation of sustainability has focused on both fiscal and institutional issues.

8.1 Impacts of the PBS Program

8.1.1 Impact of block grant spending on sector outcomes (PSIA study)

179. The Poverty and Social Impact Assessment (PSIA) study of the Bank analysed effectiveness of woreda block grant spending on education, health, and agriculture using data on woreda level per capita recurrent expenditure, and variety of education, health, and agriculture outcome indicators between 2008 and 2011112. The key findings based on log-linear regression are as given under113:

• NER and PTR showed a significant relationship with woreda-level per capita recurrent expenditure after controlling for the effects of rural/urban divide and ethnicity. According to the estimated equations, one per cent increase in per capita recurrent expenditure on education would increase net enrolment by 0.27 per cent, and PTR by 0.22 per cent.

• Similarly, significant association was found between per capita health expenditure (recurrent) and health outcomes. The regression results showed that one per cent increase in per capita health expenditure, after controlling for capital expenditure at regional level, would increase ANC by 0.086 per cent, and deliveries by SBA by 0.27 per cent.

• In case of agriculture, one per cent increase in per capita expenditure on extension workers was associated with more than one per cent increase in yield of enset, fruit, and coffee.

180. The findings clearly suggest that woreda-level spending on education, health, and agriculture has been effective in achieving key sector outcomes114.

181. The PSIA also examined the issue of equity in decentralised spending at woreda level. In this regard the study made some important observations as follows. The benefits of woreda level spending on education and health accrued to all income levels, and per capita expenditure under the existing allocation framework favoured historically disadvantaged regions and ethnic groups. More than 50 per cent of the woredas in Gambella and 30 per cent in Benishangul-Gumz spend more than 110 per cent of the national average per capita on basic services. Similarly, four majority-Anyiwak woredas are noteworthy for receiving more public resources than all other woredas in the nation. In view of these findings, the study observed that additional resource allocation and spending in disadvantaged regions and ethnic groups appeared to help them to catch up with national level basic service delivery.

8.1.2 Progress towards achieving MDG targets

182. The positive impacts of block grants on sector outcomes (as brought out by PSIA) are expected to accelerate progress towards achieving MDGs. Keeping this in mind, the present evaluation has tracked progress towards MDGs (Goals 1 to 5) applying counterfactual analysis and using latest available data115. The findings are as follows. Of the five MDG goals, the target has been fully achieved with respect to only one indicator under Goal 4 (reduction in under five mortality rate). The progress made so far with respect to Goal 1 (eradication of poverty and hunger), Goal 2 (universal education), and Goal 3 (promoting gender equity) are on track although the targets may not be achieved by 2015. But, progress towards achieving Goal 5 (improving maternal health) has remained completely off the track. On the whole, the progress towards MDGs has been satisfactory, and it may

112 Qaiser M Khan et al (World Bank, 2014): Improving Basic Services for the Bottom Forty Percent 113 Ibid (Refer to Tables 3.1, 3.2, and 3.4) 114 There are some caveats to the empirical finding. For details refer to the study op. cit. 115 MoFED carried out the last update on MDG achievements in 2012. See MoFED (2012): Assessing progress towards the Millennium Development Goals. In this section, data have been updated up to 2013-14.

Process Evaluation of PBS-3, Final Report: Impacts and Sustainability

45

be reasonable to say that the PBS program with its support for expansion of basis services has contributed in this progress (see Annex 6 for detail analysis of progress made towards MDGs).

8.1.3 Unintended or negative impact of the PBS-3 program

183. In course of reviewing relevant documents, the present evaluation has not found any specific reference or proven records of any unintended or negative impacts of the PBS program. The only unintended negative impact of PBS-3 has been its alleged linkage with human right abuses in Gambella. Though the Inspection Panel did not find any evidence of PBS program causing alleged harms, the event heightened international scrutiny of PBS-3.

8.2 Sustainability of Basic Services

8.2.1 Fiscal sustainability

184. The key factors to determine fiscal sustainability of providing basic services are: (i) demand for basic services, given the population growth; (ii) required expenditure to support the demand based on cost of services; and (iii) availability of domestic resources to meet the expenditure. Till date, no exercise has been undertaken to assess fiscal sustainability of basic services based on these factors. Only a small study on cost of basic service delivery was conducted by IGFT-TWG in a few districts in Amhara from 12-22 October 2014116. In the absence of any available study, this evaluation has made an attempt to to assess whether basic services could be fiscally sustained in the mid term (2015-16 to 2019-20).

Table 8.1: Projected expenditure, revenue, and budget deficit

ETB billion 2015-16 2016-17 2017-18 2018-19 2019-20 1. Expenditure on basic services 153.3 171.5 189.3 209.2 230.8 2. Other expenditure 71.1 78.2 86.0 94.6 104.1 3. Total expenditure [(1) + (2)] 224.4 249.7 275.3 303.8 334.9 4. Tax revenue 158.4 174.3 191.7 210.8 231.9 5. Non-tax revenue 12.7 13.9 15.3 16.9 18.6 6. Total revue [(4) + (5)] 171.1 188.2 207.0 227.7 250.5 7. Budget deficit [(1) – (6)] 53.3 61.5 68.3 76.1 84.4 8. Budget deficit as % of GDP 4.2 4.4 4.5 4.5 4.5 Assumptions:

• Projections are made taking 2013-14 as base year for which actual expenditure and revenue data are available. The projected figures for 2014-15 are not shown here.

• Population is assumed to grow as per medium variant projection by CSA. • Per capita pro-poor spending (recurrent and capital expenditures) at national level was ETB

1,459 (USD 76) in 2013-14. It is assumed to grow at 8 per cent, which was the rate of inflation in 2013-14.

• In 2013-14, the expenditure other than pro-poor spending was ETB 58.7 billion. The same is inflated at 10 per cent per year over the projection period.

• GDP is assumed to grow at 10 per cent (growth rate was10.3 per cent in 2013-14). • Tax-to-GDP and non-tax-to-GDP ratios are assumed at 12.5 per cent (actual was 12.7 per cent

in 2013-14), and 1.0 per cent (actual was 1.2 per cent) respectively.

185. The base case projection shows that if the current fiscal conditions continue, spending on basic services for the growing population would push budget deficit (without grants) to GDP ratio from 3.8 per cent in 2013-14 to 4.2 per cent in 2015-16, and then to 4.5 per cent in 2019-20 (Table 8.1). In absolute term, budget deficit will average ETB 7.5 billion (or USD 362 million) per year over the five

116 IGFT Working Group (April 2015): Cost of Basic Service Delivery and Gap in Selected Woredas


46

year period from 2015-16 to 2019-20117. This deficit is not manageable without donors funding support unless the government can boost revenue in a big way. In 2013-14, external finance (net) bridged 52 per cent of the budget deficit. Applying the same ratio, the required donors’ funding would be around USD 150 to 200 million per year.

186. The sensitivity analysis shows that one per cent point reduction in tax-to-GDP ratio would cause budget deficit to increase by about one per cent point over the projected base case figures (Figure 8.1). Thus, it is imperative that GoE take all measures to increase tax revenue commensurate with the revenue potentials of the country’s growing economy. Under worst scenario, which combines one per cent point decrease in GDP growth rate, one per cent point drop in tax-to-GDP ratio, and one per cent point increase in rate of inflation – budget deficit will significantly shoot up jeopardising provision of basic services for the growing population. This means that GoE needs to closely monitor economic growth and stability.

Figure 8.1: Sensitivity of projected budget deficit estimates

187. The present exercise on medium-term projections suffers from a few limitations. First, in the absence of unit cost estimates, budget provisions for basic services are estimated using per capita expenditure of the base year. In all probability, unit costs of service delivery as per sector standards will be much more than the base-year per capita spending. Second, the budget provision for basic services does not include any estimates for repairs and maintenance of infrastructure and facilities. Third, in the absence of any diagnostic study on domestic resource mobilisation, the present projection has relied only on historical tax and non-tax to GDP ratios for projections.

8.2.2 Institutional sustainability

188. Since inception of the PBS program, there have been conscious efforts to institutionalise good governance system and practices, and build capacity at woreda level. The objective has been to put in place a critical system to support basic services that is both financially and institutionally sustainable. In fact, the Sub-Program B of PBS-3 aims at achieving this objective. A brief discussion follows on the progress made towards institutional sustainability at woreda level.

189. The PFM has become a deep-rooted, matured system, with several institutional strengthening measures such as roll out of IBEX in most of the woredas; streamlining of internal and external audit systems; organising PFM team; and engagement of channel one coordinator in every region. However, lack of professionals and trained staff especially for procurement, and high staff turnover

117 The ETB amount is converted to USD an exchange rate of 1 USD= ETB 20.6033, which is calculated using average monthly exchange rates during January to October 2015. The monthly exchange rates were gathered from the following website: http://www.exchange-rates.org/history/ETB/USD/G/M

4.2 4.4 4.5 4.5 4.5

5.2

5.4 5.5 5.5 5.55.8

6.36.6

7.07.3

2015-16 2016-17 2017-18 2018-19 2019-20

(Percent)Basecase

1percentpointdecreaseinGDPgrothrate

1percentpointdecreaseintax-to-GDPra1o

1percentpointincreaseininfla1on

Combinedeffect


47

undermine the system’s capability for better management of public funds. The new PFM project is expected to continue with PFM reforms and address many such issues.

190. FTA activities have been largely institutionalised through development and dissemination of guidelines and templates. Though it may take more time, efforts to impart budget literacy training would slowly bring about the kind of awareness required to make FTA effective. However, it is less likely that FTA activities at current level will be sustained in the absence of any program support. A great deal of leadership by BoFED/WoFED is required to sustain the on-going institutionalisation process. Their capacity needs to be developed.

191. The grievance redress mechanisms are already in place and functional except in Afar. Given the government’s political commitment to introduce GRMs through out the country, there is no serious sustainability risk. However, it is imperative to develop and implement a separate nationwide strategic plan and program to scale up and improve GRM services.

192. Turning to social accountability, the big challenge is how to institutionalise social accountability interventions. At present, social accountability activities are driven by SAIP, the initiatives are sector specific, and the pilots are trying to develop a new institutional mechanism to mobilise citizens around service delivery. Mooted question is whether social accountability activities will be sustained without program support. The present evaluation feels that from the point of view of sustainability, a good starting point under ESAP-2 would have been to first activate and train the existing CBOs (e.g. parent teachers association) for social accountability activities. Also, more emphasis could have been given to involve local administration and political representatives for spearheading the SA movement. 193. A critical element to support basic service delivery is to establish a structure for results based management. It is imperative therefore to strengthen MIS in basic sectors. This is what MfoR has been pursuing with the sector ministries. Good progress has been made with the education and health sectors and work is in progress in the other sectors. 194. In sum, the PBS program has made significant strides towards institutionalising public financial management, citizen engagement, and grievance redress mechanisms. However, more reforms are still required, especially for institutionalisation of social accountability initiatives.

8.3 Summary

Impacts and sustainability

Impact of block grant spending on sector outcomes (PSIA study): This impact study reveals that woreda-level spending on education, health, and agriculture has been effective in achieving key sector outcomes. It also claims that benefits of the woreda-level spending have also percolated to all income levels, and per capita expenditure favoured historically disadvantaged regions and ethnic groups. Progress towards achieving MDGs: The progress towards achieving MDGs has been satisfactory except for Goal 5 (improving maternal health), which has remained completely off the track. It may be reasonable to say that the PBS program with its support for expansion of basis services has contributed in this progress. Unintended or negative impact: The present evaluation has not found any specific reference or proven records of any unintended or negative impacts of the PBS program, except the alleged link of the program with human right abuses in Gambella. This allegation was not found true, but it has put the Bank’s reputation at risk, and heightened the international scrutiny of PBS-3. Sustainability of basic services: The medium term projection (2015-16 to 2019-20) shows that if the current fiscal conditions continue, basic services to the growing population would be possible only with the donors’ funding support of USD 150-200 million per year. The PBS program has made significant strides towards institutional sustainability but more needs to be done, especially for institutionalisation of social accountability intiatives.

Chapter 9: Lessons Learned and Recommendations

This concluding chapter presents a number of lessons learned from the PBS experience, and makes a few recommendations.

9.1 Lessons Learned

195. The present evaluation draws a few lessons from the PBS experience as summarised in the discussion that follows.

196. Decentralised service delivery supported through IGFT is an effective development model to reduce poverty and achieve other MDGs. Ethiopia’s decentralised service delivery through intergovernmental fiscal transfers (block grants) to sub-national governments (regions/woredas) has been effective in making progress towards poverty reduction and achieving other MDGs. However, the progress has not been even across the regions, as well as across the income/wealth quintiles. Going forward, the development efforts need to be strengthened to promote equitable service delivery. This is what ESPES aims to achieve using PforR instrument.

197. The PBS program through its partnership and pooled-fund approach offers an effective alternative to general budget support system. The PBS program was originally designed as a response to post-2005 election crisis that resulted in suspension of GBS by some DPs. Over the years it evolved as a successful alternative to GBS, showcasing high degree of aid effectiveness, and with great potential to adopt PforR instrument.

198. Citizen engagement activities promote direct accountability of the service providers, and improve service delivery. The PBS program has extensively relied on FTA, social accountability, and GRM activities to create space for citizen engagement with local administration and service providers. The evidence is growing to show that such citizen engagement has tremendous potential to improve service delivery in all the basic service sectors.

199. Use of the country systems has been crucial for gaining the government’s ownership of the program. The PBS has made extensive use of the government’s treasury system for transfers of block grants and other program funds (excluding direct transfer of funds to the management agency). Also, the program has relied on administrative database and M&E systems for monitoring results. Such alignment of the program with the existing country systems has led to greater ownership of the program by the government.

200. A dedicated resource allocation by DPs to set up a PBS secretariat is an essential step forward to achieve donors’ coordination and harmonization. In a multi-donors set up, role of a donors’ secretariat hardly needs to be emphasised. The evidence shows that the PBS secretariat has provided a wide range of services to support program implementation as well as to achieve donors’ coordination and harmonization.

201. Pilots should be undertaken with a perspective plan on how the experience and results of pilots will be scaled up to a national level. The experience and learning of LIG pilot under PBS-1&2 was not utilised to improve woreda level investments and capacity building. Similarly, social accountability activities have been piloted in over 25 per cent woreda in the country, but a concrete blueprint is still missing for institutionalization of the citizen’s engagement in local administration of basic service delivery. There should be an end-game plan for any pilot project.

202. Decentralised basic service delivery is a complex process, and it must be supported by deep-rooted institutional reforms at sub-national level. The PBS experience suggests that for effective delivery of basic services, the local governance system (FTA, SA, GRMs) should be capable and robust, which can be achieved only through long-term institutional reforms. Both PSCAP and PBS have pursued the goal of good governance at sub-national levels. But, despite improvements over the years, there are inherent challenges (e.g. inadequate resource generation, high staff turnover, PFM weaknesses) that still impede service delivery. It is quite evident that a program like PBS alone cannot

Process Evaluation of PBS-3, Final Report: Bibliography

49

address all the governance and service delivery issues at woreda/kebele levels. The current thinking of launching a separate PFM project seems to be a move in the right direction.

203. Failure to assess environmental and social risks may put the program at reputational risk. The Bank’s Investigation Panel observed that the PBS-3 appraisal and supervision did not pay adequate attention to mitigate the potential risks arising from geographical and sector overlap of the PBS program and CDP. Although the PBS program was not found to cause harms in Gambella, the program came under heavy international scrutiny.

9.2 Recommendations

9.2.1 Focusing on education, health, and agriculture sectors where program impacts are more

204. Given that PBS support for block grants finances only recurrent expenditure, the sectors that require more recurrent expenditure have been benefited by the program. These sectors are education, health, and agriculture extension services. The capital-intensive sectors like road and water sectors have not really benefited from the PBS program. It is recommended that the PBS program focus only on education, health, and agriculture sectors, and avoid thinly spreading resources and programmatic efforts for the road and water sectors that basically need funding support for capital investment and operating expenditure.

9.2.2 Setting up a high-level steering committee for strategic discussion and decision making

205. As mentioned in this report, the semi-annual JRIS forum has not been designed to push high-level strategic discussion on issues such as regional inequities; gender mainstreaming; resource mobilisation; and institutionalisation of citizen engagement. In view of this, GoE and donors may like to set up a high-level program steering committee (PSC) involving concerned line Ministers, senior government officials, and DPs. Actual structure and functional modality needs to be elaborated once the stakeholders agree to set up a PSC. Given the strategic nature, PSC discussion needs to reflect on diagnostic data (e.g. disaggregated gender data published by CSA), findings of the analytical studies, emerging evidence from field mission, etc.

9.2.3 Strengthening donors’ harmonization

206. Harmonization framework: The present evaluation proposes a basic framework of donors’ harmonization, which needs to be elaborated further in consultation with the DPs (Table 9.1).

207. At the sector level (Level-1), harmonization will involve developing a single integrated plan and budget for the entire sector, including all DP funded activities/programs. This is currently followed in the health sector based on HSDP harmonization manual, and it may be adopted in the other basic sectors too118. The “one plan, and one budget” approach has advantage of eliminating duplication of efforts and sub-optimal use of resources.

208. The expected results of the sector plan and budget allocations needs to be laid down in a common results framework, which can be tracked jointly by the line Ministry and the DPs. At present, DPs track their individual results framework. This is a duplication of efforts and may be avoided.

209. It is further proposed that instead of having separate joint review missions by each DP, efforts should be made to have a single, combined JRM for the sector program, which will reduce burden of the line ministry and reduce transaction costs. The line Ministry and DPs will be jointly responsible for implementation of the single, harmonized plan. It would be useful to bind them through a memorandum of understanding (MoU) that describes the mutual roles and responsibilities of the signatory partners.

118 See HSDP IV, Annual Performance Report, EFY 2006, p 61


50

210. At the national level (Level-2), harmonization will involve synchronization of multi-sector programs with the same or complementary types of interventions so as to gain from synergies of the programs supported by different DPs. To illustrate the point, reference could be made to AfDB’s newly proposed program called basic services transformation program (Jan 2016 to December 2018), which consists of two components viz. (a) enhancing quality and equity in basic services; and (b) strengthening accountability, transparency and citizen’s engagement. Clearly, BSTP resembles ESPES/PBS-3. In such case, the purpose of harmonization will be to ensure that both the programs complement each other, and the activities are not duplicated. For instance, in promoting citizen engagement through FTA, SA, and GRMs, the AfDB’s program can use the tools, formats, and guidelines already developed and implemented under the PBS program. Similarly, the program implementation areas (woredas/kebeles) may be divided in such a way that the combined coverage of PBS-3 and BSTP is maximised. To achieve this kind of program synchronisation, it is imperative to develop a common strategy and work plan.

Table 9.1: Proposed framework of donors’ harmonization

Level of harmonization Approach Focus areas Responsibility

Level-1: Harmonization of DPs at sector level

Developing “one plan, one budget, and one report”

Preparation of a mid-term plan and budget based on the sector strategies and policies

• Ministry • DPs

Harmonization of donor funded programs within the same sector plan and budget

Developing a single integrated results framework

Monitoring results through a combined JRM by all DPs and Ministry

Level-2: Harmonization of DPs at national level

Synchronization of multi-sector programs with same or complementary interventions

Developing a common strategy and work plan

• DPs • SDS

A nodal agency to harmonize and coordinate with DPs This has already been done. The PBS secretariat has been reconstituted as SDS with a set of objectives & activities that inter alia would lead to better donors’ harmonization.

Functioning of nodal agency as per ToR (refer to Table 9.2)

211. Objectives and staffing structure SDS: The PBS secretariat has been reconstituted as SDS with effect from July 2015, which is now entrusted with the responsibility of donors’ harmonization. It is proposed that SDS focus on four strategic objectives (Table 9.2): (i) coordination and harmonization of the DPs; (ii) results management (M&E); (iii) supporting risk management and application of environment and social safeguards; and (iv) communication and knowledge management.

212. Given the four strategic objectives of ToR, SDS must have the required technical skill sets. It is proposed that SDS ensures service of the following experts: (i) economist; (ii) PFM specialists; (iii) safeguard specialist; (iv) M&E expert; and (v) knowledge and communication expert. The services of these experts may be availed through full-term or short-term consultants. DPs may decide the actual mix keeping in view the budget allocations.


51

Table 9.2: Proposed objectives and activities of the newly created SDS

Strategic objectives Key activities 1. Coordination & harmonization of

the DPs engaged in basic service sectors

• Coordinate and organize quarterly PSC meetings, annual JRM for strategic discussion

• Participate, and facilitate participation of DPs, in sector specific TWGs & relevant TWGs in DAG

• Prepare an update on the proceedings of the TWGs and share the same with all DPs

• Coordinate submission of data to AMP by DPs as per schedule given in the manual

2. Supporting results management (M&E)

Sector-wide activities • Develop a common results framework (CRF) in collaboration with

the DPs mainly for tracking progress towards SDG as per GTP II targets [progress against CRF will be discussed in annual JRM]

• Participate, and facilitate DP participations in sector JRMs • Develop a medium-term work plan (in consultations with DPs,

MoFED, ministries, govt. agencies, and management agency) for conducting analytical and evaluation studies (in-house and through consultants), and implement the same PBS/ESPES related activities

• Provide administrative and logistic support for holding JRIS of PBS/ESPES

• Provide technical support e.g. data analysis, analytical studies, update results framework after data validation and reconciliation of discrepancies, drafting Aide Memoire

• Facilitate DP participation in JRIS 3. Supporting risk management, and

application of environmental and social safeguards

• Provide need based technical support to the World Bank and other DPs in the areas of risk management and application of environmental and social safeguards

• Facilitate DP participation in the DAG’s special missions to assess violation of human rights (e.g. DfID’s participation in DAG’ visits to CDP sites)

4. Communication & knowledge management

• Preparation and dissemination of communication materials for: − Creating more public awareness about government initiatives to

improve access and quality of basic services; − Sharing country’s progress towards achieving SDGs; − Sending common messages in public interest; etc.

• Develop a web-based repository of knowledge products (e.g. reports, studies, success stories, best practices) for easy public access

• Develop a web-based forum for sharing knowledge products, as well as for independent exchange of views among various stakeholders

9.2.4 Benchmarking service delivery

213. In the context of quality service delivery, there is no escape from benchmarking the basic service delivery at woreda and the service provider levels. The findings of the field mission undertaken in this study has revealed that the regions have willingness and some experience of assessing service delivery at woreda level. The field mission has also observed that the sector bureaus in some of the woredas are tracking and grading the service providers. All these are encouraging trends and should be supported.

214. It is recommended that the regions and woredas be encouraged to introduce appropriate system for results based performance management (RBPM). MoFED may take the initiative to develop guidelines, tools, and templates for RBPM in consultation with the regions/woredas. It would be also necessary to conduct training programs for the BoFED and the sector bureau staff on how to use the templates of RBPM.


52

215. In addition to above, it would be extremely useful to document best practices on service delivery and use of service standards, and share them with the regions, woredas, and sectors.

9.2.5 Conducting beneficiary survey

216. Beneficiary feedback is an important input to improve service delivery and must be tracked periodically. This is a missing link in the program management and evaluation system of the PBS program. The present evaluation strongly recommends introduction of yearly beneficiary survey as an integral part of the program’s M&E system. Each region should undertake beneficiary survey using standard tools, and the regional results should be integrated to get a national perspective.

9.2.6 Regions to check five per cent administrative data related to basic services

217. It is almost a truism in every developing country that the database forming the administrative MIS is not adequately checked for accuracy and data validation. This makes it imperative to conduct a minimum level data check before using official data for assessing progress against results indicators. It is recommended that each region undertake a 5 per cent check of the sector data based on a sample of two to three woredas. CSA may provide technical support to the regions. The process may continue with rotation of sample woredas for strengthening the MIS system.

9.2.7 Expediting the study on service cost estimation and resource mobilisation

218. The medium term projections undertaken in this study shows that the government would require donors funding support to meet fiscal deficits if basic services are to be scaled up to serve the growing population. This means that federal and regional governments are urgently required to estimate costs of basic services, and the resources required to meet the costs. The federal/regional governments are also required to look for means and ways to augment resource generation. This is a long overdue exercise, and any further delay in completing this exercise would be detrimental to fiscal sustainability of basic services.

9.2.8 Institutionalization of social accountability activities

219. Pending the development of a concrete plan for institutionalization of social accountability, the present evaluation would like to make a few suggestions as follows.

• It is recommended that the regions launch sustained campaign to encourage the citizens to demand their rights for basic services and seek accountability of the service providers. Such campaign should be steered by the elected council members and local administrators at the regional, woreda, and kebele levels. The involvement of council members and administrators would instill confidence among the citizens about the government’s commitment to ensure transparency and social accountability.

• Simultaneously, the council members, the government officials, and the service providers need to be sensitized to respect the citizens’ right to demand basic services, and reminded of their duties and obligations to meet the citizens’ demand. They are also required to learn various social accountability process and tools.

• The social accountability process also needs to be embedded in the exiting community based organisations (CBOs), such as parent teacher associations at school level, health boards, water committees, and community participation in rural road construction. The CBO members need to be trained on SA process and tools

220. It is also recommended that ESAP be harmonised with similar kinds of projects or programs funded by other DPs. For instance, United States Agency for International Development (USAID) is in the process of designing a flagship program titled strengthening participation and accountability for community engagement (SPACE) leveraging on the momentum created by ESAP-2. So, it provides a good opportunity to pull resources and develop a joint program.


53

Select Bibliography

Africa Development Fund (July 2012): Appraisal Report – PBS 3

Aide Memoires, Joint Review and Implementation Support (JRIS) Missions, PBS 3 Program, Federal Democratic Republic of Ethiopia

Alem, Demmelash, Xavier Furtado, and Samuel Mulugeta (May 2011), Woreda Block Grant Transfer Formulas: Current Practices and Challenges

B Piper (2010): Ethiopia Early Grade Reading Assessment Report. RTI /USAID Ethiopia International

Central Statistical Agency (March 2015): Ethiopia – Gender-Disaggregated Development Data

_______ (March 2015): Report on Overall Data Quality Assessment for Education Sector in Ethiopia

_______ (October 2014): Report on Data Quality Assessment for Health Sector in Ethiopia

_______ (February 2012): Final Report on Data Quality Assessment for Education and Rural Road Sector in Ethiopia

_______ (September 2011): Ethiopia Data Quality Assessment (EDQAF)

DAG (July 2015): Profiles of 41 Development Partners in Ethiopia

______ (June 2014): Post-Busan Monitoring Results for Ethiopia

_______Annual Report 2013-14

Department for International Development (2014): Annual Review – Summary Sheet

_______ (2013): Annual Review – Summary Sheet

_______ (7 November 2012): Business Case for Promotion of Basic Services in Ethiopia (2012-2017)

Development Assistance Group Ethiopia (July 2010): Aid Management and Utilisation in Ethiopia – A Study in Response to Allegations of Distortion in Donor-Supported Development Programmes.

_______ (2012): Understanding VfM in the Protection of Basic Services Programme.

_______ (2013-14): Annual Report – Development Effectiveness in Ethiopia.

Draft workshop report (28 January 2016) on Services Delivery Harmonization

Ecorys Macro Consortium (25 June 2012): Ethiopia - Multi-Annual Review of PBS Program (Final Report)

Education Statistics Annual Abstract, 2013-14

Ethiopian Agricultural Transformation Agency: Annual Report 2013-2014

Fletcher Tembo (September 2013): Rethinking social accountability in Africa: Lessons from the Mwananchi Programme

Federal Democratic Republic of Ethiopia: Growth and Transformation Plan (FY 2010 – FY 2015)

Federal Ministry of Health (FMOH): Annual Performance Report, 2013-14

Health and Health Related Indicator, EFY 2007


54

House of Federation (January 2012): The Federal Budget Grant Distribution Formula 2012/13 – 2016/17

Humberto Lopez (August 2004): Pro-Growth, Pro-Poor: Is There a Tradeoff? WB Policy Research

International Bank for Reconstruction and Development and International Development Association (January 31, 2015): Managing Report and Recommendation in Response to the Inspection Panel Investigation Report, PBS 3

IPE Global (November 2011): Evaluation of Pilot Local Investment Grant

ITAD (February 2015): Value for Money Assessment of the Promotion of Basic Services (PBS) Programme (Final Report)

Management Agency (Feb 2014): FTA-RSAP-2 Linkage Workshop

_______ESAP-2 (9 May – 6 June 2013): Baseline Survey Report

_______ ESAP, Medium-Term Priorities and Roadmap (Final Draft Version)

_______ ESAP-2: Concept Note for Bridging Amendment

_______ ESAP-2: Quarterly Progress Report 14, April-June 2015

Marito Garcia, Andrew and Sunil Rajkumar (World Bank, 2008): Achieving Better Service Delivery Through Decentralization in Ethiopia

Ministry of Civil Service (2013): Woreda and City Administrations Benchmarking Survey V (Synthesis Report): Survey of Service Delivery Satisfaction Status

Ministry of Finance and Economic Development (2015): Public Financial Management – Integrated Report

_______ (September 2013): Financial Transparency and Accountability Implementation Assessment Report

_______ (21 June 2013): Protecting Basic Services, Phase II Government’s Implementation Completion Report.

_______ (March 2012): Ethiopia’s Progress towards Eradicating Poverty - An Interim Report on Poverty Analysis Study (2010/11)

_______ (2012): Assessing Progress towards the Millennium Development Goals

_______ (2004): Ethiopia: Fiscal Decentralisation Strategy

PBS Secretariat (March 2015): PBS 3: B3 Managing for Results (MfR) Component - Status Update

_______ (February 2015): Fourth Annual Report (January-December 2014) and Indicative Work Plan (2015)

_______ (2015): Supporting a New Phase of Decentralized Basic Service Delivery – Roadmap Paper for a Harmonized Approach

PEFA Final Report, April 2015

Qaiser M. Khan, et al: Improving Basic Services for the Bottom Forty Percent – Lesson from Ethiopia, The World Bank Group.

Report of Workshop on PBS 3 – Effectiveness, Equity and Sustainability Reviews, September 30, 2014


55

RTI International (September 2014): Ethiopia 2014, Early Grade Reading Assessment – Report of Findings

UNDP: National Human Development Report (2014) - Accelerating Inclusive Growth for Sustainable Human Development in Ethiopia

UNFPA (December 2012): Trends in Maternal Health in Ethiopia - Challenges in Achieving the MDG for Maternal Mortality

UNICEF: Countdown to 2015

United Nations: MDG Report 2014: Assessing Progress in Africa toward the Millennium Development Goals

Wendmsyamregne Mekasha Zike and Bereket Kelemu (2015): State of Service Delivery in Ethiopia Primary Schools: Findings from the Ethiopia Education Service Delivery Indicator Survey, Middle Eastern & African Journal of Educational Research, Issue 16

World Bank (31 January 2015): Management Report and Recommendation in Response to the inspection Panel Investigation Report

_______ (19 August 2015): Project Appraisal Document (PAD) for Enhancing Shared Prosperity through Equitable Services

_______ (April 2015): Public Finance Management Assessments: Integrated Report Based on Public Expenditure Financial Accountability (PEFA) Framework

_______ (January 2015): Ethiopia Poverty Assessment, Poverty Global Practice, Africa Region

_______ (20 July 2015): Restructuring Paper

_______ (21 November 2014): Investigation Report, PBS 3 Project (P128891)

_______ (20 December 2013): Implementation Completion and Results Report, Protection of Basic Services Program Phase II Project

_______ (17 October 2013): General Education Quality Improvement Project II

_______ (29 August 2012): Project Appraisal Document (PAD) for Promoting Basic Services Phase 3 Project

_______ (29 August 2012): Country Partnership Strategy

_______ (30 June 2010): Implementation Completion and Results Report, Protection of Basic Services Project

Young Lives (Ethiopia Policy Brief 1, 20 December 2012): Improving education quality, equity and access in Ethiopia – findings from the Young Lives school component

Process Evaluation of PBS-3, Final Report: Annex-1

56

Annex-1: Terms of Reference

Title: Process Evaluation: Promoting Basic Services Phase 3 (PBS 3) Program in Ethiopia

1. INTRODUCTION & BACKGROUND

Introduction

1.1 On behalf of the Government of Ethiopia and donors to the Promoting Basic Services (PBS) Program the PBS Secretariat wishes to commission evaluation specialists (the “Evaluation Provider”) to design and implement an independent process evaluation of the Promoting Basic Services Phase 3 Program (PBS 3) in Ethiopia.

1.2 This evaluation is part of the PBS 3 Core Evaluation series comprising of four components:

a) Process Evaluation: To provide an independent assessment of the performance of the PBS program including identification of strengths and weaknesses which will lead to recommendations for improvements to the current phase and any possible successor program

b) Demand Side Service Utilization Study: A quantitative analysis on service utilization using Benefit Incidence Analysis (BIA) will be employed to assess service utilization by households, disaggregated by socio-economic status (SES), rural/urban location, gender, and other socioeconomic characteristics such as elderly, squatters, vulnerable children, disabled, etc.

c) Beneficiary Incidence Analysis: A qualitative (participatory) assessment to identify the percentage of people responding positively on whether their views are incorporated during the design/planning phase of service delivery program (Perception survey results).

d) Consolidated Core Evaluation Report: Drawing on overall evaluative evidence from core evaluation and other analytical products.

1.3 This process evaluation will focus on the policy and operational effectiveness of PBS 3119. PBS 3 plans a total of two process evaluations during the course of the program. These Terms of Reference cover the first evaluation only. The evaluation will draw on available evidence and data, including other evaluative and analytical studies concerning the program’s effectiveness and impact e.g. Poverty and Social Impact Analysis, Beneficiary Impact Assessment Study, DfID Value for Money Study.

Background

1.4 Since 2006, the Protection of Basic Services (PBS) program has supported the delivery of five basic services across Ethiopia: health, education, water and sanitation, agriculture, and (starting in 2009) rural roads. Simultaneously, the program also promotes stronger citizen engagement and improved accountability through local civil society organizations. Based on a strong partnership between Ethiopia’s major donor agencies and its federal, regional, and district-level administrations, the program’s support has been essential to achieving several of the MDGs in Ethiopia.

119 Definition used for this assignment: Process Evaluation assesses and explains the particular program operations and processes. It can be used throughout the life of a program, from start-up to maturity, and whenever program operations change thereby requiring adjustments of the program. During program start-up, process evaluation is used to monitor initial program implementation so that bottlenecks can be addressed speedily and good practice can be documented for future applications. After program stabilizes and matures, a process evaluation can be used to provide ongoing feedback to management. Unlike impact evaluation, process evaluation cannot establish causality with a known margin of error. Process evaluation can only uncover plausible reasons why a program is working or not.


57

1.5 PBS 3 has now been effective since December 2012. It continues to support the promotion of better services in Ethiopia, including ensuring sustained and further gains in access paired with an increased focus on quality of services. The program now comprises two sub-programs:

o Sub-program A: Basic Service Block Grants (USD 6.2 billion). This sub-program l finances recurrent (salaries, operations and maintenance) expenditures in the five basic service sectors (education, health, agriculture, water and sanitation, and rural roads) at local levels. The Basic Service Block Grants are disbursed from the federal level to the regions, and from the regions to the woredas. They are discretionary at the sub-national level, i.e. each woreda decides the allocation of the grants to the five basic service sectors based on the needs in their woreda.

o Sub-program B: Strengthening Local Accountability and Transparency Systems (USD 114.6 million). While the country’s decentralized, integrated system of fiduciary, administrative and information governance is robust, the Government and Development Partners recognize that it can and should continue to be strengthened, particularly in its implementation in the large number of very diverse woreda administrations throughout Ethiopia. Sub-program B will seek to maintain and strengthen these systems, with a strategic focus on the woreda-level as the front-line of administration with responsibility for PBS objectives. The sub-program includes three interrelated components: (B1) Citizen’s Engagement, (B2) Local Public Financial Management, and (B3) Managing for Results.

1.6 With strong government ownership (and financing) and effective coordination and harmonization of donor support, the PBS program has contributed to impressive development results. Examples include:

• a reduction in the under-5 mortality rate from 123 per thousand in 2005 to 88 per thousand in 2010;

• a dramatic increase in school net and gross enrolment rates; and • improvement in the share of the rural population with access to clean water from 35 per cent

in 2004/05 to more than 60 per cent in 2009/10.

1.7 To track progress towards common objectives and agreements, the program is based on a set of interrelated "Core PBS Principles". These core principles derive from Government commitments reflected in its Constitution, its current Growth and Transformation Plan (GTP), as well as sectoral and multi-sectoral plans. These Core Principles provide an important backdrop to understanding the scope of the process evaluation and include:

• Effectiveness: The Effectiveness principle focuses on how to maintain effective service delivery with a view to identifying ways to further improve them. Although effectiveness is influenced by a broad range of issues, the PBS program considers adequate sectoral resource allocation, balanced intra-sectoral allocation, and results achieved as it reviews the Effectiveness

• Sustainability. The PBS program seeks to ensure that financing of basic services can be sustained over the long-term even without the PBS Program. When reviewing sustainability, the PBS program considers financing sources for decentralized basic services, and the unit costs of basic service delivery.

• Additionality. The Additionality Principle ensures that government priority to the MDGs are expressed in medium-term commitments to increase overall financing for the federal block grants and hence to basic service at sub-national level and that allocated resources for basic services are flowing in a predictable manner.

• Fairness. The Fairness Principle ensures that resource allocations from the federal government to the regions and from regional governments to woredas are rules-based and


58

transparent, and that block grant disbursements to the regions as well as from regions to woredas are executed as planned in budget allocations

• Equity. To review the Equity Principle, PBS will track and assess any discrepancies in access to basic services among the different regions and woredas and between genders, with a view to identifying possible interventions to address these discrepancies

• Transparency. A core principle of the PBS program's contribution to the access and quality of basic services is to provide stakeholders with more information about resource flows, standards and results.

• Fiduciary Probity. The PBS program relies on a robust fiduciary system reaching from the federal level through regions to local administrations. It involves numerous mechanisms to maintain the strength of that system.

• Predictability. The predictability principle seeks to ensure mutual accountability by predictable resource flows for basic service delivery results. Development Partner contributions need to be based on longer-term commitments, so that agreed disbursements can be made on time. Likewise, the Government has a responsibility to accurately reflect these DP contributions

1.8 Aid effectiveness has been a hallmark of the PBS program since its inception. By way of its multi-dimensional approach, the program uses country systems while supporting systems-strengthening in the areas of monitoring & evaluation and public financial management. Through the PBS Secretariat, the program has dedicated human and financial resources to ensure donor coordination and harmonization resulting in improved joint oversight, more effective dialogue, and reduced transaction costs for all parties.

1.9 As PBS 3 approaches its Mid Term Review and looks towards the emergence of a possible successor program, this evaluation will be important to informing ongoing implementation, and future program design aspects.

1.10 A full description of the program is available in the World Bank’s Project Appraisal Document available at http://documents.worldbank.org/curated/en/2012/08/16711203/ethiopia-third-phase-promoting-basic-services-project.

2. PURPOSE AND OBJECTIVES

Purpose

2.1 The main purpose of the process evaluation will be to provide an independent assessment of the performance of the PBS program including identification of strengths and weaknesses. The evaluation will focus on the interaction between policy and operational effectiveness. Information generated by this evaluation will be important to (i) provide feedback to enhance ongoing program implementation (ii) enhance program accountability and (3) inform decision making amongst government and development partners in future program design and resource allocation.

2.2 The specific objectives of the evaluation are to (i) review the performance of the project across the dimensions of relevance, effectiveness, efficiency, impact, sustainability and coherence (ii) provide recommendations on future improvements for program design.

2.3 In assessing the performance of the program the evaluation will consider the following dimensions, based on the OECD DAC Criteria for Evaluating Development Assistance120:

a. To evaluate the relevance of PBS 3 in the following context:

120 http://www.oecd.org/development/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm


59

• To what extent are the objectives of the program still valid? How appropriate is PBS 3 in view of the political, economic and social context in Ethiopia?

• Are the activities and outputs of the program consistent with the overall goal and the attainment of its objectives? How appropriate are the Core Principles of the program in this regard?

b. To evaluate the effectiveness of PBS 3 in the following context: • To what extent are program objectives likely to be achieved? This should have a particular

focus on operational and policy impacts of PBS, especially in the areas of equity and fairness.

• What are the major factors influencing the achievement or non-achievement of the objectives?

c. To evaluate the efficiency of PBS 3 in line with DFIDs Value for Money approach and in the following context:

• Have PBS 3 objectives been reached in a timely manner? • Could the same results have been achieved with fewer resources? Could the same resources

have been used differently to achieve more results? d. To evaluate the impact of PBS 3 in the following context: • What has happened as a result of PBS 3 until the present stage of implementation?

e. To evaluate the sustainability of PBS 3 investments in the following context: • To what extent has PBS helped to promote improved service delivery financing, also

differentiated by different revenue sources? • To what extent has PBS 3 led to a better understanding of domestic resource mobilization

achievement and potential? • To what extent has PBS 3 enhanced service delivery capacity at decentralized levels e.g.

through participation, empowerment, ownership, local resources are available and sustained political support exists.

f. To evaluate the coherence of PBS 3 in following contexts: • How does PBS 3 complement existing government strategies for service delivery in terms

of national planning and sectoral engagement? • To what extent have development partners harmonized and aligned common efforts towards

PBS 3? In particular to examine the role of the PBS Secretariat in facilitating any such actions.

Constraints and Limitations:

2.4 A number of constraints should be noted when addressing the above objectives.

• The evaluation will draw on a rich foundation of available information and data. The evaluation will triangulate existing information and rely greatly on key informant discussions, rather than collecting or commissioning new data.

• The scope of certain objectives is narrowed given the particular nature of PBS 3. For example (i) the evaluation of impact is reduced given the challenges of attribution under the program e.g. national coverage, nature of recurrent expenditure. A range of analytical studies will help inform this objective. (ii) the evaluation of sustainability reflects specific work objectives set out during program implementation.

• Certain dimensions of analysis will have greater priority than others. The process evaluation fills knowledge gaps especially relating to relevance, effectiveness, sustainability and coherence.

Recipients


60

2.5 The immediate recipient of this evaluation will be the PBS Secretariat. The main users will be the Government of Ethiopia and Development Partners.

2.6 The evaluation will also be used to provide feedback and information to all users of basic services in Ethiopia. Relevant communications will need to be developed for a range of audiences as part of the communication and learning strategy of the evaluation.

3. DELIVERABLES AND OUTPUTS

3.1 The overall deliverable of this exercise is a finalized process evaluation to be delivered during calendar year 2015.

3.2 In arriving at this deliverable the following outputs will be expected:

Output Content Timeframe, Quality Assurance

Format

Inception report

(i) draft methodology (ii) literature and data to be reviewed, and any information gaps arising (iii) stakeholders to be consulted (iv) regions to be visited (v) schedule and (vi) challenges of evaluabitliy, mitigation measure and any resulting changes to the Terms of Reference

Week 1 Submitted to Evaluation Steering Group (ESG) for approval

Word File, 20 pages

Methodological Framework

Matrix of evaluation questions, and evaluation framework

Week 2 Submitted to Evaluation Steering Group (ESG) for approval

Word or Excel File

Mid-term report (i) progress of consultancy (ii) preliminary findings (iii) specific challenges to be addressed

Week 6 ESG to provide comments

Word File, 20 pages

Draft PBS Review Document & Presentation

Final draft of process evaluation Presentation to ESG, other stakeholders

Week 9 ESG to provide comments

Word File, 80 pages, PowerPoint (25 slides)

Final process evaluation

Final process evaluation, with full recommendations Presentation to PBS 3 Stakeholders

Week 11 Word File (80 pages) Annexes Power Point (25 slides) Master data file

3.3 It should be noted that in addition to this exercise, a follow up evaluation will be conducted during the calendar year 2017. The preference is for the same firm to undertake this evaluation. However the PBS Secretariat reserves the right to advertise for this work, particularly if the evaluation provider does meet performance expectations.

4. METHODOLOGY

4.1 At the outset of the exercise the evaluation provider is expected to set out a work-plan with detailed methodology.

4.2 To produce the above deliverables and outputs the evaluation provider is expected to undertake certain tasks, including (but not limited to) the following:

• review existing project evaluations and studies undertaken in each of the components and the five basic service sectors


61

• collect and analyze both qualitative and quantitative information related to program; in particular materials generated under the Effectiveness, Equity and Sustainability reviews

• familiarize themselves with available supervision materials, mainly in the form of supervision mission reports and Aide Memoire’s from Joint Review and Implementation Support (JRIS) missions

• Participate in a JRIS mission as observers • Undertake semi-structured individual interviews based on clear guidelines and check lists.

Stakeholders interviewed will include key government and development partner stakeholders involved in PBS. These should include but not be limited to: ministers; relevant sector ministry staff at national, regional and district levels; academics; and private sector and civil society resource persons.

• Undertake group interviews / focus groups. Organisation of one or more thematic meetings or workshops with representatives of the different categories of stakeholders in order to collect the same type of information from different persons at the same time, to increase its validity and the overall understanding of the issue. Focus groups with members of private sector associations and other relevant sectors of civil society based on clear guidelines and check lists should be ensured.

• Share initial data analysis with relevant stakeholders through the organisation of a workshop so as to ensure completeness and correctness

• Others as suggested.

4.3 The evaluation provider should be aware that this stocktaking exercise will proceed in tandem with design and appraisal process for the next phase of the project by the World Bank. As such, the evaluation provider may be expected to share interim ideas and conclusions to help inform the ongoing design process.

4.4 As previously noted, the evaluation is expected to draw heavily from the regular monitoring data emerging from national and sector survey and administrative sources and parallel evaluations and studies. A range of additional studies are also ongoing which will increase the depth of the evidence base including a formal impact evaluation of the social accountability component.

4.5 A large part of the evaluation work, especially the work concerning PBS inputs and their effects on public policy processes and management is likely to be carried out in Addis Ababa. However in order to ensure a full local perspective and to substantiate findings via triangulation the evaluation provider will execute field visits at regional and woreda levels to consult with government officials, beneficiaries, civil society and private sector. .

4.6 The PBS Secretariat will work with the evaluation provider to compile relevant documentation and agree on appropriate stakeholders for interview. If the evaluation team is not permanently based in Ethiopia the team should make necessary arrangements for a visit to conduct the necessary interviews including with stakeholders based outside Addis Ababa.

5. LEVEL OF EFFORT, SKILLS AND QUALIFICATIONS

5.1. The maximum total duration of the assignment is 170 working days, over no more than a four month time period. This time period includes preparation, travel, field visits, reporting, submission of comments and submission of final report. Two visits to Ethiopia will be required if the evaluation provider is based outside of the country.

5.2. It is expected that the evaluation provider will mobilize three senior experts (including team leader) and one junior expert. They are expected to work as follows: Expert 1 (50 days), Experts 2 and 3 (35 days) and Expert 4 (50 days).

5.3. This assignment will require a team of experts with skills and expertise in a range of academic disciplines including development, evaluation, public financial management,


62

economics, social accountability, financial transparency, grievance redress and systems-strengthening/capacity building. Knowledge of PBS sector areas is also expected. The team shall be gender balanced with a mix of international and Ethiopian consultants.

5.4. The evaluation provider shall determine the precise skill mix of the team. Indicative profiles for each team member are as follows:

- Senior Expert 1: Economist/Evaluator and Team Lead, with knowledge of service delivery

• University graduate, minimum of Master’s Degree in Economics or related field and a minimum of 10 years of professional experience in one of more of the following areas: program reviews, project design, decentralisation, aid effectiveness and service delivery.

• Demonstrated capacity in the design of complex multi-sector evaluation systems for programs in developing countries

• Knowledge of PBS sectors and service delivery, including supply and demand side factors that promote overall service delivery outcomes.

• Knowledge of new and innovative aid instruments • Demonstrated ability to manage and work as a team and to work with a large range of

national and local stakeholders • Ability to communicate and present (both orally and in writing) complex evaluation issues in

simple terms in a succinct and diplomatic way;

- Senior Expert 2: Governance or Public Policy and Administration Expert

• University graduate, minimum of Master’s Degree in Political Science, Public Policy, Law, Economics, International Development or Governance and Public Administration with a minimum of 10 years of professional experience in related areas: decentralisation, aid effectiveness, service delivery, citizens’ engagement.

• Excellent knowledge of basic services and public financial management issues in developing countries and associated programming in these areas

• Proven experience in undertaking similar type of reviews, evaluations, and lessons-learned preferred.

- Senior expert 3: Evaluation specialist, with sectoral expertize in human development areas such as health, education and agriculture.

• University graduate, minimum of Master’s Degree in Economics, Statistics, Monitoring and Evaluation, or related studies, and a minimum of 10 years of professional experience in monitoring and evaluation, project design, decentralization, and service delivery.

• Expertize in human development sectors including health, education and agriculture. • Expertise in gender, socio-economic and poverty analysis

- Junior expert 3: Economist researcher

• University graduate with minimum of Master’s Degree in Economics or social science and a minimum of 5 years of professional experience in project management and research work;

• Proven research and data management expertise; • Knowledge on decentralisation, national legislation and institutional set up in Ethiopia • Expertise on monitoring and evaluation of large government-led programs will be an

advantage;

5.5. In addition to the above individual skills, the overall evaluation team should have (i) excellent writing, editing and communication skills (If the team proves unable to meet the level of quality required for drafting the required outputs, the evaluation provider will supply, at no


63

additional cost, an immediate technical support to the team to meet the required standards); (ii) working knowledge of Amharic; (iii) knowledge of World Bank implementation approaches; (iv) demonstrated understanding of the political economy of Ethiopia; (v) relevant academic skills for the methods proposed and (vi) familiarity with PBS.

6. DUTY OF CARE

6.1 The evaluation provider is responsible for the safety and well-being of their personnel (as defined in Section 2 of the Contract) and third parties affected by their activities under this contract, including appropriate security arrangements. They will also be responsible for the provision of suitable security arrangements for their domestic and business property. All duty of care, transport, translation and logistical support, office space, and insurances will be the responsibility of the evaluation provider.

6.2 DFID will share available information with the supplier on security status and developments in-country where appropriate. DFID will provide the Duty of Care risk assessment matrix for Addis Ababa.

6.3 All evaluation provider personnel will be offered a security briefing by the British Embassy/DFID on arrival. All such personnel must register with their respective embassies to ensure that they are included in emergency procedures. A copy of the DFID visitor notes will be provided (and a further copy each time these are updated), which the supplier may use to brief their personnel on arrival.

6.4 In addition, all members of the evaluation team will take the security briefing from the UN Department of Security Services (UNDSS) before travelling outside of Addis Ababa.

6.5 The evaluation provider is responsible for ensuring appropriate safety and security briefings for all of their personnel working under this contract and ensuring that their personnel register and receive briefing as outlined above. Travel advice is also available on the FCO website and the evaluation provider must ensure they (and their personnel) are up to date with the latest position.

6.6 The evaluation provider is responsible for ensuring that appropriate arrangements, processes and procedures are in place for their personnel.

6.7 If the evaluation provider is unwilling or unable to accept responsibility for Security and Duty of Care as detailed above, bids will be viewed as non-compliant and excluded from further evaluation.

6.8 Acceptance of responsibility must be supported with evidence of capability and DFID reserves the right to clarify any aspect of this evidence. In providing evidence evaluation providers should consider the following questions:

a) Have you completed an initial assessment of potential risks that demonstrates your knowledge and understanding, and are you satisfied that you understand the risk management implications (not solely relying on information provided by DFID)?

b) Have you prepared an outline plan that you consider appropriate to manage these risks at this stage (or will you do so if you are awarded the contract) and are you confident/comfortable that you can implement this effectively?

c) Have you ensured or will you ensure that your staff are appropriately trained (including specialist training where required) before they are deployed and will you ensure that on-going training is provided where necessary?

d) Do you have an appropriate mechanism in place to monitor risk on a live / on-going basis (or will you put one in place if you are awarded the contract)?

e) Have you ensured or will you ensure that your staff are provided with and have access to


64

suitable equipment and will you ensure that this is reviewed and provided on an on-going basis?

f) Do you have appropriate systems in place to manage an emergency / incident if one arises?

7. LOGISTICS AND PROCEDURES

7.1 The evaluation provider will not be required to deploy on a permanent basis. The evaluation work-plan should detail how long the provider will be in Ethiopia.

7.2 All evaluation team members are expected to commence their assignment at the same time.

7.3 The evaluation providers are asked to provide details on the ethical issue they expect to face (including potential risks to interviewees) and mitigating measures that they will apply to this task, in particular the issue of engaging with children and young people.

7.4 Inception, work planning and review meetings with PBS Secretariat staff and the Evaluation Steering Group will take place as required to ensure that the evaluation team has all the necessary advice and guidance to ensure that the Government of Ethiopia and Development Partners as customers for this work are satisfied with the various measures in place.

7.5 Opportunities and timescales for commenting on a quality assurance of study outputs will all be agreed at the outset.

8. GOVERNANCE, REPORTING & CONTRACTING ARRANGEMENTS

8.1 The evaluation provider will be accountable to an Evaluation Steering Group (ESG), led by the PBS Coordinator – Task Team Leader (TTL) for this exercise. The steering group should include members of the Government of Ethiopia, Donor Partners and the PBS Secretariat.

8.2 The target audience for the evaluation reports will be this Steering Group in the first instance, but a wide range of other interested stakeholders thereafter. These are likely to include both Ethiopian and international interest groups, civil society groups, academics and donors. The evaluation team will need to ensure that results are accessible to all these groups. In particular, users of basic services will be both part of the process of monitoring and evaluating, and key audiences in terms of sharing final analyses.

8.3 The terms of reference for the ESG will be developed before the final contract is signed.

8.4 The ESG, through the PBS Secretariat structure, will facilitate the provision of documents for the evaluation provider’s work.

8.5 Each deliverable will be reviewed and quality assured by peers, experts and government counterparts.

8.6 Evaluation providers must ensure that they behave in an ethical manner. In particular, suppliers must not ask local operators to enter into an exclusivity agreement.

8.7 Financial disbursements will be made upon satisfactory completion of agreed activities within the evaluation work-plan and framework.

8.8 It is essential that the evaluation report be presented in Ethiopia to government, development partners and other core PBS stakeholders. It is envisaged that the Joint Review and Implementation Support (JRIS) mission will be a key mechanism for dissemination of the key findings but the evaluation provider should also consider alternative mechanisms of disseminating relevant information to the full range of relevant stakeholders. In this connection, evaluation provider will prepare a dissemination strategy.

8.9 English will be the language of all correspondence, reports and presentations.


65

8.10 Within 10 working days of receipt of each deliverable, the ESG will provide written comments to the evaluation provider. The evaluation provider will submit finalized version of all reports, within the next five (5) working days. The PBS Coordinator will approve all final outputs.

8.11 The evaluation provider will submit ten (10) hard copies and one electronic copy (CD-ROM) to the PBS Coordinator, TTL.

8.12 Visa for Ethiopia must be obtained prior to commencement of the mission.


66

Annex-2: List of People Met

People interacted at federal level

Organisation Name of the Person Met Designation The World Bank Dr Subrata Dhar PBS Coordinator

Wendmsyamregne Mekasha Senior Social Protection Specialist

Samuel Mulugeta Economist and Member IGFT Getahun Tafese Desta Consultant- Social Protection Demelash Demssie Operations Officer Alex Kamurase Senior Social Protection Specialist TameneTiruneh Senior Social Protection Specialist Colin Andrews TTL- Senior Social Protection Specialist Girma Woldetsadik Operations Officer Roman Tesfaye Consultant – Health Anne Bakilana TTL – Health Section Qaiser Khan Sector Lead Economist and Donor Co-chair

Department for International Development

Ahmed Mohammed Governance Advisor Jyoti Tewari Senior Human Development Adviser Belay Addise Education Adviser Martha Solomon WASH Adviser Betseit Sisay Results & Programme Manager Bobby Stansfield Human Development Policy Advisor Berhanu Lakew Economist

African Development Bank Mulumebet Merhatsidk Senior Socio-economist Italy Development Cooperation Wanda Grimaldi M&E Office Austrian Development Cooperation Astrid Wein Head of Coordination Office

European Union Clara Molera Donor Co-chair Ministry of Education Bereket Kelemu Ayele Director, EMIS & ICT Directorate Ministry of Health Noah Elias Director, Policy and Planning Directorate Ministry of Water, Irrigation and Electricity

Nuredin Mohammed Beyan

Director, Water Supply and Sanitation Directorate

Shewanesh Demeke, PBS Coordinator Central Statistical Agency Aberash Tariku Deputy Director General

Esayas Muleta Director, Data Quality and Standards Coordination

Ministry of Finance & Economic Development

Getachew Negera Head, COPCO Musa Mohammad EMCP Director Melaku Kifle Senior Program Specialist and Advisor Tiruwork Bekele PBS 3 Team Leader Belay Asrat Senior Social Accountability Specialist Aklilu Tekle Sr Public Finance Specialist Dereje Kassaw PBS Accountant


67

People interacted at regional level

Name Designation/Institution Oromia

Getu Busa Director

Tesfaye Gemechu PBS Coordinator

Etefa Abetu Accountant Mulugeta Aseffa FTA Focal Person Terefe Disasa BOA Deputy Bureau Head Kebede Tabor BOE Budget Expert Gemechu Asfaw BOH Procurement and Finance process owner Hirpho Rural Road Authority Procurement and finance process owner

Benishangul Gumuz Getachew Disasa Channel 1 Program Coordinator Melkamu Desalegn BOE Planning and Programming Department Head Tewodros Abera BOE Planning Expert Fekadu Regasa Planning and Programming process owner Tarekegn Wodi M&E Expert Habte Weldeyesus BOA Planning and Programming Process Owner Mihretu Mekonnen BOA M & E Expert Addisu Abate BoWater Planning and Programming Process Owner Beyene Alemu Rural Road Planning and Programming process owner

Somali Kalif Aden Coordinator, Channel One and BoFED Aden Ali PFM Coordinator, BoFED Mustefie Yosouf GIS Case Coordinator, BoFED Abdurahiman Mohammed Inspection and Internal Audit Process Owner, BoFED Abdurahim Ahmed Budget and Planning; and M&E Process Owner Anbes Getahun Bekele Audit Manager, ORAG Deh Abdulahi Muhied Planning, M&E, HMIS, Resource Management Process Owner, Health Bureau Mustefie Muxumed Cigal Commissioner, GRM Kedir Mohammed Planning Expert, BOE Farsel Siyaad Planning expert, BOE Beshir Seherheil RMSN Coordinator, BOE Dey Abdi Aden Planning and Budgeting Expert, BOW Ahmed Abdi Omer WASH and M&E Expert

SNNPR Tarekegn Nuramo Channel One Coordinator, BoFED Zeleke Legesse BOE Expert Mandefro Mathewos BOE Expert Meharu Yohannes Data Analyst, BOW Esubalew Assefa Planning Expert, Rural Road Mohammed Abdela DPPM, Rural Road Aknaw Kawza Malefia DPPM, BOH Reta Abayneh Inspection and Audit Process Owner, BoFED Fikre Tamire Planning Officer, ORAG Girum Zerihun Good Governance Officer, GRM


68

Name Designation/Institution Tesfahun Behiru FTA Focal Person, BoFED Mubarek Awel Planning, M&E Expert, BoFED

People met at woreda level

Name Designation/Institution Oromia Chelia

Lemma Wakgari Head, WoFED Lechisa Abdisa Deputy Head, WoFED Desalegne Gedefa Purchaser Dirbi Adugna Gafesu Head, Woreda Agriculture Office; and Vice President, Woreda Admiistration Motuma Tolesa Deputy Head, Woreda Agriculture Office Gezahigne Fekadu Deputy Head, Woreda Rural Road Office

Jima Arjo Tamiru Dheressa Head, WoFED Fetene Tadessa Head, Woreda Agriculture Office; and Vice President, Woreda Administration Biyazinilign Belay Deputy Head, Woreda Water Office Asmara Diriba Woreda Water Supply Expert Mekonen Abera Head, Woreda Rural Road Office

Benishangul Gumuz Asosa

Eliyas Abdi Head, WoFED Zewdu Belay IAU Team Leader Abdurahman Ousman Head, Agriculture Woreda Office Ahmed Seid Head, Woreda Water Mines and Energy Office Ahmed Mekonen Water Resource Development Coordinator Asherif Yonis Head for Woreda Rural Road Office Andualem Nadew Woreda Construction Foreman

Bambasi Abdu Jalaladin Head, WoFED Zeleke Head, Finance Department Fikremariam Yeshewalul Head, Woreda Agriculture Office

Atsbeha Kasu Water Supply Process Owner Tagay Endale Woreda Construction Followup and Control Expert Wondwosen Tamiru Community Participation and Information Expert Ashenafi Bambasi ESAP2 Coordinator

Somali Jigjiga

Ahmed Salah Hussien Head, WoFED Abdi Omer Abdi Head, Agriculture Office Mohammed Shiek Hassen Head, Health

Matios Mebrate Health Office Expert Yousuf Ahmed Nur Auditor


69

Name Designation/Institution Isse Abdi Muhhamed Agriculture Office Desk Ali Abdilah Hussien Expert Ahmed Nour Mohammed Wereda Aministration Expert

Kedir Mohammed Planning Expert, BOE Faysel Siyad Planning Expert, BOE Beshir Seherdid RMSN Coordinator Deyr Abdi Aden Planning and Budgeting Expert Ahmed Abdi Omer WaSH M&E Specialist Abdurzak shiek Curriculum Desk Head Fatuma Abdrahma Water and Road Expert

Shinile Hassen Muhumod Head, Water Office Fetih Shukri Mohamod Head, Health Office

Rahma Head, WoFED Fahmi Mussie Water Office Expert Forhon Ebrahim EPI Expert Abdilwadud Abdi Head, Woreda Education Office

Abdulfetah Mohammed

Curriculum Development and Input Dissemination and Examination Process Owner, Shinile Woreda Education Office

Fuad Galab Supervisor, Shinile Woreda Education Office Abdi Aden Kaynaan Wereda Administrator

SNNPR Dilla Zuria

Ashenafy Halchye Head, WoFED Nebyu Kore Deputy Head, WoFED and DPPM Kebede Fotu Head, Audit Desk Shebru Werera Head, Procurement Desk Dawit Jigso Wereda Administrator Temesgen Bekele Head, Rural Road Office Mekonen Kebede Kebele Connection, Rural Road Office Expert Berhanu Degela Head, GRM Unit; Wereda Administration G/Mariam Gobena Former GRM head, wereda Administration Ychenek Worera WoA planning Head Haimanot Ararso Drinking water core process coordinator Melese Fikadu Education Office Teaching Learning Core Process Head Abinet Demeke Education Office Teachers and Employees Administrative Core Process Head Wossen Woldemariam Health Office Deputy Bureau Head

Bekele Sele Education Office Head Tamrat Education Office Planning Officer

Shebedino


70

Name Designation/Institution Zewdie Kebede Head, WoFED Deneke Darimo WoFED, DPPM Thomas Namaro Rural Road, Kebele Connection Expert Legesse Sarmo Rural Road, Road security Expert Netesha Yenter Procurement Officer, FTA Focal, WoFED Yonas Health Office Planning Head Dawit Desalegn Health Office Planning Officer Asmra Head, WoE Degfie debesa Planning Head, WoE Urago Husien Head, Water Office Hanchamo Younkura Woreda Administrator Demsie Chale Water Planning Head Taye Lebese Head, Good Governance and Local Affairs Office Abdela Husien Agriculture Planning Head


71

Annex-3: Evolution of the PBS Program

The PBS program has evolved from one phase to another, leveraging on the lessons learned in each phase. Besides, the program design in each phase gave due considerations to emerging trends in basic service delivery, unfinished agenda, and the challenges in program implementation. In this sense, the PBS program has progressed through a dynamic process. The tables below summarises the evolution of the PBS program through phases.

Comparative program designs

PBS-1 (3.5 years) PBS-2 (3.5 years) PBS-3 (5 years) Basic Information

(June 2006 – December 2009) (May 2009 – January 2013) (January 2013 – January 2019) USD 2,958 million (completion) USD 3,763 million (completion) USD 1216 million (revised budget)

Risk rating: Substantial Risk rating: Moderate Risk rating: High Program Development Objectives

• Protecting and promoting the delivery of basic services by the sub-national governments

• Deepening transparency and local accountability

• Expanding access & improving the quality of basic services

• Deepening transparency and local accountability in service delivery

• Expanding access & improving the quality of basic services by funding block grants that ensure adequate staffing and operations

• Strengthening the capacity, transparency, accountability and financial management of sub-national governments.

Coverage of sectors Education, health, agriculture, water supply & sanitation

Coverage of sectors Education, health, agriculture, water supply & sanitation, rural roads

Coverage of sectors Education, health, agriculture, water supply & sanitation, rural roads

Components 1. Block grant for basic services 2. Local investment grant (LIG) 3. Health MDG support facility 4. Financial transparency and

accountability (FTA), including M&E, and capacity building

5. Social accountability (SA)

1. Block grant for basic service 2. Local investment grant (LIG) 3. Health MDG support facility 4. FTA & SA 5. Monitoring and Evaluation

1. Block grant for basic service 2. Strengthening local

accountability & transparency systems • Citizen's engagement (FTA,

SA, GRM) • PFM and procurement • Managing for Results

3. Results enhancement funds (later dropped)

Core Principles 1. Additionality 2. Fairness 3. Fiduciary responsibility 4. Accountability

1. Sustainability in additionality (S)

2. Accountability and fairness (A)

3. Fiduciary standards (F) 4. Effectiveness (E)

1. Effectiveness (review) 2. Sustainability (review) 3. Additionality (test) 4. Fairness (test) 5. Equity (review) 6. Transparency 7. Fiduciary probity 8. Predictability

Review Process 1. JRIS/JBAR (initially quarterly,

later half-yearly) 2. Pre-JRIS field visits

1. JRIS/JBAR (half-yearly) 2. Pre-JRIS field visits

1. JRIS/JBAR (half-yearly) 2. Pre-JRIS field mission with

thematic focus


72

Transferring learning from one phase to another

PBS-1: Lessons learned PBS-2: New features Rural roads are important as they help communities to access social services while also improving their ability to participate in productive economic activities.

Additional sector of rural roads (community access roads) included as a basic service sector. [Reflected in sub-program A of PBS II]

Capacity constraints evident in the decentralized delivery of services, especially at lower levels.

Efforts to strengthen capacity, especially at woreda level stepped up as part of Sub-Program C of PBS II. For example, strengthen accounting and oversight systems.

Need for greater consideration of local context in design of social accountability initiatives

PBS II took into account existing accountability systems and their regional variation; local comfort zones and entry points; local socio-political context; and an increased focus on promoting local level innovations. [Reflected in Sub-Program C]

Changes in the risks relative to Phase I necessitated changes in principles and performance assessment framework for the next phase.

The tests and shared commitments re-focused to embody a “SAFE” approach.

PBS-2: Lessons learned PBS-3: New features Need for stronger results focus Sub-component B3 (Managing for Results) and

sub-program C (Results Enhancement Fund), included, latter one got dropped

Need for streamlining program implementation arrangements and simplifying program structure

Program streamlined into just two components.

Requirement of efforts to promote opportunities for citizens to get information about service delivery as well as provide feedback

Strengthened the links and synergies between the financial transparency and accountability and social accountability sub-components.

Note: Lessons learned are drawn from the Bank’s ICR for PBS-1 (June 30, 2010) & PBS-2 (Dec 20, 2013).


73

Annex-4: Progress Towards Equitable Services in Education and Health

This annex presents an assessment of progress made towards achieving equitable services in primary education, and maternal and child health care services.

A. Primary Education

Regional level inequity: The regional data on NER, GPI, PTR, and proportion of qualified teachers as of 2013-14 reveal that some regions are lagging behind the others. For instance, NER (Grade 1-8) is less than 85 per cent in Afar, Dire Dawa, and Bennishangul-Gumuz (Figure 1). Similarly, primary schools in Somali, Oromiya, and SNNPR are having PTR higher than the norm of 50 students per teacher. In the same vein, the proportion of qualified teacher at Grade 1-4 is less than 50 per cent in Afar, SNNPR, and Somali. So, for improving equitable primary education, specific interventions would be necessary in the aforesaid regions.

Figure 1: Regional inequities in terms of select education indicators (2013-14)

Inequity across wealth quintiles: EDHS provides estimates of net attendance ratio (NAR) of primary school-age children (7 to 12 years) across wealth quintiles of the households. A close scrutiny of EDHS data reveals significant variations in NAR across the wealth quintals. The gap in NAR between poorest and richest wealth quintiles was 43 per cent points in 2005, reduced to 32 per cent points in 2011, and stayed around that level in 2014 (Figure 2).

Similar kind of wealth disparities in respect of other critical indicators of primary education (e.g. drop out rate, completion rate) is likely to exist. It is imperative therefore to conduct periodic household

6078 83 85 88 94 97 100

118 120136

(A)NER,Gr1-8(percent)Source:MoEdata

85

54 5243 41 39 34 33 30 27 26

(C)Pupil-teacherra1o,Gr1-8(percent)Source:MoEdata

0.8 0.9 0.9 0.9 0.9 0.9 1.0 1.0 1.01.2

1.3

(B)GPI,Gr1-8(percent)Source:MoEdata

2331 34

58 60 61 61 63 6572

96(D)Qualifiedteachers,Gr1-4(percent)

Source:MoEdata


74

survey to gather more evidence on inequities across wealth quintiles so as to inform operational policy and strategies aiming at equitable school education.

Figure 2: Net attendance ratio in poorest and richest wealth quintiles

B. Primary Health Care

Regional level inequity: This evaluation has assessed regional inequity in terms of select indicators viz., population per health post, per health centres, and deliveries by skilled providers. The findings are follows.

Population-to-health post ratio is higher than the standard (5000 populations per health post) in Dire Dawa, Tigray, Harari, and Amhara, indicating shortage of primary health facilities (Figure 3). These regions plus SNNPR and Somali also have shortage of health centres vis-à-vis the standard (25,000 populations per health centre). Contrary to common belief, the emerging regions seem to have adequate health posts and health centres relative to population size. Turning to deliveries, Dire Dawa has highest proportion of deliveries assisted by skilled professionals (58 per cent) followed by Harari (40 per cent). SBA deliveries are much lower in the remaining regions and least in Afar (7 per cent).

Figure 3: Regional inequities in terms of select health indicators (2013-14)

Poorestquintile

68.2

83.9 81.5

25.4

52.0 48.7

2005 2011 2014

(Percent)Source:EDHS2005,2011,2014

Richestquin1le

Poorestquintile

2,617

3,685

4,558

4,766

5,134

5,241

6,150

7,030

7,522

14,194

BenishangulG

Gambella

Afar

SNNPR

Somali

Oromiya

Amhara

Harari

Tigray

DireDawa

(A)Popula1onperhealthpostSource:ComputedusigMoHdata

14,103

23,957

25,340

26,372

26,975

28,717

29,000

29,333

31,406

38,943

BenishangulG

Gambella

Afar

SNNPR

Somali

Oromiya

Amhara

Harari

Tigray

DireDawa

(B)Popula1onperhealthcentreSource:ComputedusingMoHdata


75

Inequity across wealth quintiles: Like in many developing countries, coverage of maternal and child health services are higher in the richest quintile compared to the poor quintiles. For instance, the gap in ANC coverage between women in the poorest and in the richest wealth quintiles was 45 per cent points in 2005, increased to 58 per cent points in 2011, and then marginally reduced to 51 per cent points in 2014 (Table 1). Similarly, gap in delivery by skilled providers in the poorest and richest wealth quintiles increased from 26 per cent point in 2005 to 50 per cent points in 2014. The poor-rich gap in Penta-3 coverage also increased from 22 per cent points in 2005 to 36 per cent points in 2011. On the whole, inequity in maternal and child service coverage has increased over the years.

Table 1: In equities in coverage of maternal and child health services

Per cent

2005 2011 2014

Poor

est q

uint

ile

Ric

hest

qui

ntile

Diff

eren

ce

Poor

est q

uint

ile

Ric

hest

qui

ntile

Diff

eren

ce

Poor

est q

uint

ile

Ric

hest

qui

ntile

Diff

eren

ce

Contraceptive prevalence rate a 4 37 33 13 52 39 27 60 33

ANC by skilled providers 13 58 45 17 75 58 23 75 51

Delivery by skilled providers 1 27 26 2 46 44 5 55 50

Penta-3 vaccine coverage 26 48 22 26 62 36 EDHS data not available a Any method, married women Source: EDHS

In conclusion, the analysis of secondary data clearly shows that despite progress in primary education and primary health care services, regional level inequities still exist at significant level. Inequities between poorest and richest wealth quintiles are also pronounced in these two sectors. It is in this context, the World Bank’s new program ESPES aims to ensure that the whole population benefit from basic services, particularly the bottom 20 per cent of income groups and most poorly performing woredas.

7 9 9 10 12 14

24 24

40

58(C)Deliveriesassistedbyskilledproviders(percent)Source:EDHS,2014


76

Annex-5: PBS-3 Results Data (Achievements vs. Targets)

A. Achievements in Basic Sectors

Education Table 1: Achievement of targets in the education sector

Indicators Data Source

2010-11

(Base Year)

2011-12

2012-13 2013-14

Tar

get

Act

ual

Tar

get

Act

ual

Higher-level indicator

1. NER, Grade 5-8 (%) EMIS 47.3 48.1 74.0 47.2 80.0 49.5

2. GPI, Grade 5-8 (%) i EMIS 0.96 0.99 0.98 0.98 0.99 0.98

PDO level indicator

3. Student-teacher ratio, Grade 5-8 (%) ii EMIS 44.2 41.5 40.5* 37.8 45.0* 36.4

4. Qualified teachers (%) iii EMIS 47.4 60.0 72.0 64.7 83.0 70.2

Intermediate result indicator

5. Qualified teachers (no. in ‘000) iv EMIS 141.7 195.1 175 216.7 215 258.3

Note: The targets are as per the revised results framework. * The target figures for 2012-13 and 2013-14 as shown in the results framework are not consistent. i. Computed using GER data from Education Statistics Annual Abstract, 2013-14 ii. Computed dividing number of enrolled students at Grade 5-8 by total number of teacher at that grade. The

data are collected from Education Statistics Annual Abstract, 2013-14. The estimated figures are different from the student-teacher ratios reported in the Annual Abstract and the presentations made at April 2015 JRIS.

iii. The per cent of qualified primary teachers (Grade 1-8) having diploma and above are sourced from presentation at April 2015 JRIS, which essentially represent MoE data.

iv. Computed multiplying number of primary teachers (Grade 1-8) by per cent of qualified teachers. The statistical abstracts do not report the absolute number of qualified teachers.

Figure 1: Availability of primary school teachers

64.2 60.8 62.7 59.4

44.2 41.5 37.8 36.4

2010-11 2011-12 2012-13 2013-14

(A)Student-teacherra1o(no.)Source:ComputedusingMoEdata

Grade1-4 Grade5-8

20.930.4

43.854.7

84.5 90.8 92.0 91.7

2010-11 2011-12 2012-13 2013-14

(B)Qualifiedteachers(percent)Source:MoEdata

Grade1-4 Grade5-8


77

Health Table 2: Achievement of targets in the health sector


2010-11

(Base Year)

2011-12

2012-13 2013-14

Tar

get

Act

ual

Tar

get

Act

ual


1. Penta-3 vaccine coverage (%) i HMIS 84.7 84.9 89.0 87.6 94.0 91.1

DHS 57 NA NA NA 62 NA

2. ANC during 1st visit (million) ii HMIS 2.40 2.55 2.43 2.86 2.48 2.52

PDO level indicator

3. Population per HEW (no.) iii HMIS 2,107 2,170 2,500 2,334 2,500 2,449


4. Health extension worker (no. in ‘000) iv HMIS 38.3 36.7 34.0 38.0 36.0 35.9

Note: The targets are as per the revised results framework. NA: Not available i. Source: HSDP IV, Annual Performance Report, EFY 2005 & EFY 2006 ii. Source: Presentation in April 2015 JRIS iii. Source: Presentation in April 2015 JRIS iv. Source: Aide Memoire, 27-30 April, 2015

Agriculture Table 3: Achievement of targets in the agriculture sector

Indicators Data source

2010-11

(Base Year)

2011-12

2012-13 2013-14

Tar

get

Act

ual

Tar

get

Act

ual


1. Major food crops productivity (Quintal/ha) i CSA 16.5 17.0 19.2 17.8 20.5 20.3

PDO level indicator

2. Direct beneficiaries of agriculture extension services (million) ii MoA 9.05 10.51 11.40 11.66 12.80 13.09


3. DA with specialization in agro-ecological zones and high value crops (no.) iii MoA 1,466 1,850 3,201 2,170 3,800 2,811

Note: The targets are as per the revised results framework. NA: Not available i. Source: GTP Annual Progress Report 2014 & Aide Memoire, 27-30 April, 2015 ii. Source: Presentation in April 2015 JRIS iii. Source: Presentation in April 2015 JRIS. The DA specialisation includes: irrigation; fruits and vegetables; and coffee,

tea and spices.


78

Water and Sanitation Table 4: Achievement water and sanitation targets


2010-11

(Base Year)

2011-12

2012-13 2013-14

Tar

get

Act

ual

Tar

get

Act

ual

Higher-level indicator 1. Rural water supply coverage within 1.5 km

radius (%) i MoWE 48.9 55.2 80.0 a 66.5 86.0 a 75.5 MoWE 48.9 55.2 61.0 b 66.5 66.0 b 75.5

2. Households using a latrine (%) ii DHS 61.7 NA 66.5

Survey PDO level indicator 3. Non-functional water schemes (%) iii MoWE 25.5 24.0 14.0 17.2 12.0 15.5 Intermediate result indicator No indicator Note: The targets are as per the revised results framework. NA: Not available i. Source: GTP Annual Progress Report 2014 & Presentation in April 2015 JRIS.

a. Revised GTP targets b. Revised MoWE targets

ii. The figures are estimated using DHS data. The open defecation per cent is deducted from 100 to arrive at the figure of households using some form of improved or non-improved facility. It may be further noted that April 2015 JRIS did not discuss results against this indicator.

iii. Source: Presentation at April 2015 JRIS

Rural Roads Table 5: Achievement of targets in the road sector


2010-11

(Base Year)

2011-12

2012-13 2013-14

Tar

get

Act

ual

Tar

get

Act

ual

Higher-level indicator 8 Time to nearest all-weather road (hour) i ERA 3.4 2.9 2.0 2.1 1.6 1.8 PDO level indicator 9 Roads in fair and good conditions (%) ii ERA 81 86 84.6 86. 85.9 86 Intermediate result indicator 10 Woreda desks with appropriate staff (no.) iii ERA 155 500 390 500 470 500 Note: The targets are as per the revised results framework. NA: Not available i. Source: GTP Annual Progress Report 2014 & Presentation at April 2015 JRIS ii. Source: GTP Annual Progress Report 2014 & Presentation at April 2015 JRIS iii. Source: Presentation at April 2015 JRIS


79

B. Achievements under Sub-Program B

Table 6: Achievement of FTA targets


2010-11

(Base Year)

2011-12

2012-13 2013-14

Tar

get

Act

ual

Tar

get

Act

ual

PDO level indicator

1. Citizens who are informed about woreda budget (%) MoFED 19 19 21 29 23 29


2. Woredas that post refined and simplified budget and expenditure templates (%) MoFED 70 92 80 96 90 97

3. Basic service units that post the standardized service delivery templates (%) MoFED 25 40 30 55 40 69

Note: The targets are as per the revised results framework. Source: Aide Memoire, 27-30 April, 2015

Table 7: Achievement of social accountability targets


2010-11

(Base Year)

2011-12

2012-13 2013-14

Tar

get

Act

ual

Tar

get

Act

ual

PDO level indicator

1. Citizens who report that woreda officials have actively sought the views of people in their kebeles on improving quality of basic services (%) i

MA 48 49 50 36 50 36


2. Woredas where SAIP facilitate the use of SA tools (no.) ii MA 50 NA 100 223 150 223

3. SAIP woredas that have developed joint action plans for service quality improvement based on interface meetings between service users and providers (no.) iii

MA NA NA 70 70 120 218 a

Note: The targets are as per the revised results framework. MA: Management Agency NA: Not available a As of 30 June 2015, a total of 250 joint action plans have been developed in 218 woredas i. Source: Aide Memoire, 27-30 April 2015. See the comments in the text about the validity of the reported data. ii. Source: ESAP-2: Concept Note for Bridging Amendment 5, p 5 iii. Source: ESAP-2: Concept Note for Bridging Amendment 5, p 5


80

Table 8: Progress in achieving GRM targets


2010-11

(Base Year)

2011-12

2012-13 2013-14

Tar

get

Act

ual

Tar

get

Act

ual


1. Regions with functioning GRMs consistent with the best international practice (no.) Survey 2 2 5 5 6 6

2. Regions preparing and adopting standardised manual and guidelines for GRMs MoFED 3 3 6 5 8 8

Note: The targets are as per the revised results framework. Source: Aide Memoire, 27-30 April 2015.

Table 9: Progress in strengthening PFM and procurement at woreda level


2010-11

(Base Year)

2011-12

2012-13 2013-14

Tar

get

Act

ual

Tar

get

Act

ual

PDO level indicator

1. Number of woredas producing IBEX and IFMIS (no.) MoFED NA 935 734 892 800 800


2. Audit coverage by ORAG (%) MoFED 30 32 35 45 40 40

3. Audit coverage by OFAG (%) MoFED 90 96 92 100 95 100

4. Woredas that implemented prior period audit recommendations (%) MoFED 5 NA 20 14.9 25 5.1

5. Woredas having procurement specialists with regional procurement training (no.)

5 50 NA 100

All

wor

edas

in

3 re

gion

s a

6. Woredas that post all procurement award decision 5 NA 200 NA 400 254

7. Regions submit quarterly and annual IFR to COPCO without major inconsistencies (no.) COPCO 4 NA 7 10 8 11

Note: The targets are as per the revised results framework. NA: Not available a SNNPR, Amhara, and Dire Dawa claim to have procurement specialists in all woredas. Source: Aide Memoire, 27-30 April, 2015


81

Table 10: Progress under MforR component


2010-11

(Base Year)

2011-12

2012-13 2013-14

Tar

get

Act

ual

Tar

get

Act

ual


1. PBS sectors with their indicators assessed through implementation of EDQAF (no.) CSA 2 0 2 1 2 2

2. Woredas that rolled out EMIS (no.) MoE 0 0 300 NA 600 817 a

3. Health centres implementing HMIS (no.) MoH 72 80 80 87 90 99

4. Woredas rolled out WASH MIS MoWE 50 58 300 58 600 330

5. Regions implementing MoA MIS MoA 0 0 4 0 8 0

6. Regions implementing ERA MIS ERA 0 0 4 0 8 0

Note: The targets are as per the revised results framework. NA: Not available a The figure is quoted from DfID (2014): Annual Review, p 20. The results table given in Annex 3 in April 2015 Aide Memoire contains no figure against the target. Source: Aide Memoire, 27-30 April, 2015


82

Annex-6: Progress Towards Achieving MDGs

The present evaluation has tracked progress towards MDGs (Goals 1 to 5) using counterfactual analysis and based on latest available data. The findings are summarised in this annex.

Eradication of extreme poverty and hunger (Goal 1): The proportion of population below the poverty line declined sharply from 44.2 per cent in 1999-00 to 27.8 per cent in 2011-12 (Figure 1-A). This represents a whopping reduction by 17.7 per cent points over a period of one-and-a-half decade. A further reduction by 3.8 per cent points would enable the country achieve MDG target of 24 per cent. Given that inflation is now contained to single digit, the MDG target has possibly been achieved by this time. However, it should be borne in mind that even if poverty reduction touches 24 per cent that would mean 22 million population still remaining below the poverty line.

Figure 1: Reduction in poverty and hunger

The food poverty or hunger, as measured by proportion of underweight children (under 5 year), has also declined from 47.2 per cent in 2000 to 28.7 per cent in 2011-12 (Figure 1-B). If the trend continues, it is quite likely that MDG target of 22.7 per cent underweight children will be achieved in 2015. On the whole, Ethiopia is on track to achieve MDG targets of poverty and hunger reduction.

Universal education (Goal 2): NER increased sharply from 52.2 per cent in 2001-02 to 85.4 in 2011-12 and continued almost at the same level, and then rose to 92.6 per cent in 2013-14 (Figure 2-A). It is quite likely that NER will reach quite close to MDG target of 100 per cent.

Figure 2: Net enrolment ratio (Grade 1-8)

While NER is on track, there are two caveats to note. First, NER at first cycle (Grade 1-4) always remained significantly higher than NER at second cycle (Grade 5-8). The gap was 35.4 per cent points in 2005-06, and it increased to 58.4 per cent points in 2013-14 (Figure 2-B). The consistently lower NER at second cycle reflects dropouts before reaching the second cycle. Another disturbing trend is the persisting difference between GER and NER. At first cycle, difference was as high as 44.6 per

24

44.2

38.7

29.6 27.8

15

25

35

45

55

1999-00 2004-05 2010-11 2011-12 2015

(A)Popula1onbelowpovertyline(percent)Source:MoFED,MDGReport,2012

22.7

47.2

38.4

28.7

15

25

35

45

55

2000 2005 2011 2015

(B)Underweightchildren(percent)Source:DHS,2000,2005,2011

100

52.2

77.585.4 85.7

92.6

30

50

70

90

110

2001-02 2005-06 2011-12 2012-13 2013-14 2015

(A)NER(percent)Source:MoEData

73.092.2 95.3

107.9

37.648.1 47.2 49.5

2005-06 2011-12 2012-13 2013-14

(B)NERbycycle(percent)Source:MoEdata

1stCycle(1-4) 2ndCycle(5-8)


83

cent points in 2005-06, indicating enrolment of many children at higher age compared to normal schooling age of 7 years. The difference has been reduced since then but it still stood high at 29.0 per cent point in 2013-14 suggesting that enrolment of over aged children had continued.

Children from poorer backgrounds tend to enroll later, make slower progress, and are more likely to drop out without completing the schooling121. Absenteeism and drop-out among boys is often higher than for girls and are largely due to their involvement in unpaid domestic and agriculture works, whereas for girls these are mainly the result of a need to look after younger siblings and the direct cost of schooling (such as school uniforms, exercise books, pens and pencils, parental contributions towards school improvement activities, etc.)122.

Another MDG indicator to measure universal education is the primary completion rate. The progress of this indicator over the years has been completely off the track. The completion rate at Grade 5 increased from 42 per cent in 2001-02 to 76.1 per cent in 2012-13 and then fell to 69.5 per cent in the following year (Figure 3-A). The completion rate at Grade 8 had been even lower. The proportion of children completing Grade 8 increased from a low base of 18.7 per cent in 2001-02 to 52.8 per cent in 2012-13 and then declined to 46.7 per cent in 2013-14 (Figure 3-B). Though the reasons for dip in completion rates both at Grade 5 and Grade 8 are not known, it is evident that achievement in NER has been compromised by faltering progress in primary completion rate.

Gender equity in primary education (Goal 3)123: For Grade 1-8, GPI improved significantly from 0.85 in 2005-06 to 0.93 in 2013-14 (Figure 4). Though the progression is on track, achieving MDG target value of unity in 2015 is less likely124.

Figure 3: Completion rates, Grade 5 and Grade 8

Figure 4: Gender parity index (Grade 1-8)

121 See Young Lives (Ethiopia Policy Brief 1, 20 Dec. 2012): Improving education quality, equity and access in Ethiopia – findings from the Young Lives school component. 122 Ibid 123 Goal 3 also includes achieving gender equity in secondary education and participation of women in the parliament. In the context of this study, these aspects have not been analyzed in this section. 124 It may be mentioned here that gender disparities widens at secondary and tertiary levels as compared to primary level education. Early marriages, especially among children from rural and poor households, tend to inhibit educational progression among girls from primary to secondary level.

100

42.062.7

73.8 76.1

69.5

0

50

100

150

2001-02 2005-06 2011-12 2012-13 2013-14 2015

(A)Comple1onrate,Grade5(percent)Source:MoEData

100

18.741.7

52.1

52.8 46.7

0

50

100

150

2001-02 2005-06 2011-12 2012-13 2013-14 2015

(B)Comple1onrate,Grade8(percent)Source:MoEData

1.00

0.84

0.900.93

0.70

0.80

0.90

1.00

2005-06 2010-11 2013-14 2015

(Percent)Source:MoEdata


84

Reduction in child mortality (Goal 4): Ethiopia reduced U5MR to 64 per 1000 live births in 2013, and thereby achieved Goal 4 two years ahead of the deadline125. However, reduction of Infant Mortality Rate (IMR) remained slightly off-track in 2011 and the country is less likely to achieve MDG target of 31 per 1000 live births (Figure 5-A). Furthermore, death of new-born continues to be a major challenge. The new-born accounts for 40 per cent of under-five deaths in the country126.

Figure 5: Child mortality

Improving maternal health (Goal 5): At 871 deaths per 100,000 deliveries in 1990-91, Ethiopia was one of the countries with the highest Maternal Mortality Ratio (MMR) in the developing world. MMR declined to 871 in 2000-01, and to 673 in 2004-05 (Figure 6-A). A modest reversal in progress was recorded during 2010-11 when MMR increased to 676 deaths per 100,000 live births. Given the huge gap between 2011 figure and the MDG target value of 267 deaths per 100,000 live births, the progress has been completely off the track. Most maternal deaths occur during delivery and post-delivery period. This indicates that there are still significant obstacles in terms of access to and provision of antenatal and obstetric health care services, especially in rural areas. In 2014, coverage of ANC from a health professionals (which is another MDG indicator of maternal health) and proportion of births attended by skilled health personnel was as low as 41.2 per cent (Figure 6-B) and 15.4 per cent respectively127.

Figure 6: Maternal health

In conclusion, the country made satisfactory progress in achieving MDGs except for improving maternal health (Goal 5), which has remained completely off the track. It may be reasonable to say that the PBS program with its support for expansion of basis services has contributed in this progress.

125 In September 2013, MoH and UNICEF jointly announced this achievement. 126 UNICEF: Countdown to 2015, p 4. 127 Source: Mini-EDHS, 2014, p xi

63

166

123

88

20

70

120

170

220

2000 2005 2011 2015

(B)U5MR(per1000livebirths)Source:EDHS

31

9777

59

0

50

100

150

2000 2005 2011 2015

(A)IMR(per1000livebirths)Source:EDHS

267

871

673676

100

600

1100

2000 2005 2011 2015

(A)MMR(per100,000livebirths)Source:EDHS

60

26.7 27.633.9

41.2

0

50

100

2000 2005 2011 2014 2015

(B)ANCcoverage(percent)Source:EDHS

ANC

Date post:	13-Sep-2019
Category:	Documents
Upload:	others
View:	7 times
Download:	0 times

Promoting Basic Services Program (Phase-3) in Ethiopia...

Documents