The Global Evaluation of Emergency Response Funds … Glob… · · 2017-09-12The Global...

The G l oba l E va lua t ion o f E m er genc y Respo n se F un ds (E RFs)

I n c e p t i o n R e p o r t a n d W o r k p l a n

By:

Universalia Management Group

S e p t e m b e r 2 5 , 2 0 1 2

G l o b a l E v a l u a t i o n o f t h e E R F s - I n c e p t i o n a n d W o r k p l a n R e p o r t

March 2013 i

UNIVERSALIA

A c r o n y m s

CERF Central Emergency Response Fund

CHF Common Humanitarian Fund

DRC Democratic Republic of Congo

EGS Evaluation and Guidance Section

ERF Emergency Response Fund

GFDRR Global Fund for Disaster Reduction and Recovery

GHD Good Humanitarian Donorship

HC Humanitarian Coordinator

HCT Humanitarian Country Team

HRF Humanitarian Response Fund

IASC Inter-Agency Standing Committee

INGO International Non Governmental Organization

NGO Non Governmental Organization

NNGO National Non Governmental Organization

OCHA Office for the Coordination of Humanitarian Affairs

OECD Organisation for Economic Cooperation and Development

oPt Occupied Palestinian Territories

PPT PowerPoint

RBM Results Based Management

RC Regional Coordinator

TORs Terms of Reference

UMG Universalia Management Group

UN United Nations

UNICEF United Nations Childrens Fund

WB World Bank


March 2013 iii

UNIVERSALIA

C o n t e n t s

1. Introduction 1

1.1 Purpose of the Evaluation 1

1.2 Background and Context 2

1.2.1 Current ERF Activity An Overview 2

1.2.2 Our Initial Understanding of Context and Functioning of Emergency Response Funds 7

1.2.3 Stakeholder Analysis 11

2. Emerging Issues 16

2.1.1 Where do ERFs Fit? 16

2.1.2 The Critical Mass of ERFs 16

2.1.3 The Transformative Agenda 16

2.1.4 The Results of ERFs 17

2.1.5 The Overall Focus of ERF Activity 17

2.1.6 The Implications of the Country Based Orientation of the ERF Mechanism 17

2.1.7 The Life Cycle of ERFs 18

2.1.8 Gender and Disadvantaged Persons Considerations 18

2.1.9 Evaluations making a Difference 19

3. Evaluation Methodology 20

3.1 Overall Data Collection Plan 20

3.1.1 Field Work Plan 20

3.2 Field Mission Data Collection 23

3.3 Specific Methodologies 24

3.3.1 Document review 24

3.3.2 Desk Reviews 24

3.4 Additional in-person Interviews, Telephone Interviews / Focus Groups 25

3.5 Addressing Unforeseen Circumstances 25

3.6 Surveys 25

3.7 Revised Evaluation Matrix 26

3.7.1 A Provisional Theory of Change 26

4. Critical Path and Workplan 41

4.1 Schedule of Activities 41

4.2 Team and Team Assignments 41

4.3 Deliverables 45


iv

March 2013

UNIVERSALIA

C o n t e n t s

4.3.1 Overview 45

4.3.2 Deliverable 2: The Field Mission Reports 45

4.3.3 Deliverable 3: Draft Evaluation Report and Validation Workshop 45

4.3.4 Deliverables 4 and 5 Final Evaluation Report and Presentation to the Evaluation Reference Group 45

5. Coordination and Logistics 46

5.1 The Role of OCHAs Evaluation and Guidance Section (EGS) 46

5.2 The Role of Country Offices 46

5.3 Quality Assurance 47

6. Evaluability Issues 47

6.1 Uneven Availability of Data 47

6.2 The Nature of Stakeholders 47

6.3 Operations of Country Level Advisory Groups 47

6.4 The Difficulty in Monitoring ERFs 47

6.5 Local Context 48

6.6 Closed ERFs 48

7. Dissemination Strategy 49

7.1 OCHA Senior Management at Headquarters in both New York and Geneva 49

7.2 The Reference Group 49

7.3 Country Offices 49

7.4 NGO Stakeholders 50

7.5 Other UN Agencies and Donors 50

7.6 The General Public and Interested Communities 50

E x h i b i t s

Exhibit 1.1 Emergency Response Funds active during the period from 2009 to 2011 3

Exhibit 1.2 ERF Allocations by Cluster 4

Exhibit 1.3 Contributions to ERF over the last 10 years 5

Exhibit 1.4 Donorship concentration of ERFs 6

Exhibit 1.5 Resources Allocation by Organization type 7

Exhibit 1.6 The ERF Process 11

Exhibit 3.1 Emergency Response Funds Selection Criteria 21

Exhibit 3.2 Tentative Field Mission Plan and Options 23


March 2013 v

UNIVERSALIA

E x h i b i t s

Exhibit 3.3 Preliminary schedule for field missions 23

Exhibit 3.4 ERF Results Framework 26

Exhibit 3.5 Evaluation Matrix 30

Exhibit 4.1 Updated Schedule 41

Exhibit 4.2 Updated Gantt Chart 44

A p p e n d i c e s

Appendix I List of People Interviewed 51

Appendix II Internal Survey 53

Appendix III External Survey 67

Appendix IV Interview Protocol (NGOs) 77

Appendix V Interview Protocol (OCHA Staff at Country level) 79

Appendix VI Interview Protocol (UN and Donors) 83

Appendix VII Interview Protocol (OCHA HQ Staff New York and Geneva) 85

Appendix VIII Content Plan - Final Report 87

Appendix IX Content Plan - Field Mission Report 91

Appendix X Emergency Response Fund Report 93

Appendix XI Terms of Reference 99

Appendix XII Major Documents Reviewed for the Inception Report 115

Appendix XIII Quality Assurance Provisions 117


March 2013

1

UNIVERSALIA

1 . I n t r o d u c t i o n

The Universalia Management Group Limited (also known as Universalia or Universalia Management Group or UMG), is pleased to submit to the United Nations Office for the Coordination of Humanitarian Affairs (OCHA) the Inception and Workplan Report for Global Evaluation of the Emergency Response Funds (ERFs).

The report provides an update on the set of activities conducted since beginning the contract, and the decisions made regarding the evaluation methodology. This inception report responds to the agreed revised Terms of Reference (TORs) and contract, which was signed in July 2012.

Section 1 Introduction: presents purpose and scope of the evaluation and the Background and Context ;

Section 2 Issues and themes: presents the issues emerging from the Inception Phase;

Section 3 Evaluation Methodology: provides an updated methodology and evaluation matrix for the assignment including indicators and data collection and data analysis methods to be used;

Section 4 Workplan: provides a workplan and updated schedule for activities for the evaluation;

Section 5 Logistics: describes logistics and assistance required from schedule of activities;

Section 6 Evaluability Issues: Describes remaining challenges to the evaluation;

Section 7 Dissemination Strategy : Describes how the eventual Final Report will be circulated; and

Appendices: present the stakeholders interviewed during the inception mission as well, the survey and interview tools which will be used in this evaluation, and the content plans for the Field Mission Reports and the Final Report.

It should be noted that several of the elements laid out in the TORs for this Inception Report have been transferred to the Final Report with the approval of the Evaluation and Guidance Section. These include the element which would have addressed prior evaluations and that which would have given a report on ERF activity in the 15 locales (12 open and 3 closed). In both instances, Universalia was of the view that these elements are better contained in a full scale draft report as they may amplify findings.

11 .. 11 PP uu rr pp oo ss ee oo ff tt hh ee EE vv aa ll uu aa tt ii oo nn

In a protracted process such as this, there is a clear benefit to beginning an Inception Report by reprising the heart and soul of the exercise as a whole, namely the Objectives and Key Areas of the evaluation. Therefore, as per the TORs, the Objectives of the evaluation are to:

Provide an independent assessment of the contribution of ERFs to the humanitarian communitys ability to address critical unforeseen humanitarian needs in a timely and effective manner;

Examine the contribution of ERFs to strengthening the leadership of Humanitarian Coordinators, coordination role of the cluster system, and building partnerships (in particular with national and local NGOs);


2

March 2013

UNIVERSALIA

Examine the role of the elements of OCHA that plan, administer and report on ERFs, map the progress made since 2009 in improving the effectiveness and efficiency of ERFs, and identify the areas of internal strengths and weaknesses.

The TORs laid out a set of Key Areas to be examined. They are:

The success of ERFs in meeting their goal to support early action and response to critical unforeseen humanitarian needs in a timely and effective manner;

The performance, value added, and cost effectiveness of ERFs in the context of the overall humanitarian architecture and humanitarian reform process (e.g. RC/HC strengthening, cluster approach, development of effective partnership between UN and non-UN humanitarian actors, including local and national NGOs);

The level and nature of accountability between ERF recipients to RC/HC, donors, beneficiaries, and others;

The strategic, managerial and operational aspects of ERFs, including structure and focus, the efficiency and effectiveness of administration, fundraising, needs assessment, criteria for allocation of funds, monitoring and reporting, and connectedness with other funding sources;

Internal (e.g. staffing capacity, management arrangements) and external (e.g. contextual and political) factors which affect ability of ERFs to deliver on their objectives;

The accountability and risk management systems in place;

This evaluation will generate specific, actionable recommendations whose purpose is twofold:

At the global level, to inform the review of ERF Guidelines in 2013 and the development of policy in relevant areas; and

At the country level, to lead to improvements in ERF management, processes and operations.

11 .. 22 BB aa cc kk gg rr oo uu nn dd aa nn dd CC oo nn tt ee xx tt

11 .. 22 .. 11 CC uu rr rr ee nn tt EE RR FF AA cc tt ii vv ii tt yy AA nn OO vv ee rr vv ii ee ww

What is an ERF

The 20011 OCHA Annual Report sets out a concise definition of an ERF:

ERFs are country-level pooled funds managed under OCHA auspices. The HC is the overall custodian of ERFs where grants are allocated to NGOs, UN agencies, the International Organization for Migration and the Red Cross/Red Crescent Movement for response to rapidly evolving needs. The funds are un-earmarked, and decisions for fund allocations are made at the country level in a timely, flexible and coordinated manner.

The first Emergency Response Fund was established in Angola in 1997. Others have been established since then, in conditions of critical humanitarian need and willingness of one or more donors to commit assistance. In several instances varying nomenclature has been used; however the basic characteristics of such a pooled fund have remained the same. Give the relatively small size of ERFs in comparison to other pooled funding, ERF grants are small sized, mostly sub $250,000 and largely, but not exclusively directed toward NGOs.


March 2013

3

UNIVERSALIA

Current situation of the ERFs

As of July 2012 there are 12 ERFs in operation: Afghanistan, Colombia, Democratic Republic of Congo (DRC), Ethiopia, Haiti, Indonesia, Kenya, Myanmar, oPt, Pakistan, Yemen and Zimbabwe. Three of more ERFs, namely Iraq, Nepal and Uganda have been closed by end of 2011. Generally, ERFs are relatively small in size (less than $10 million), provide small to medium sized grants (less than $500,000), and predominantly fund NGOs. The first ERF, established in Angola in 1997 to respond to increasing humanitarian needs caused by years of conflict, had received $24.5 million from eight donors until its closing in 2004. The Ethiopia HRF is continuously the largest fund since the creation of ERFs and is unique as it operates in a country with no CAP and where it is the only humanitarian financing mechanism. The Haiti/ERF, with the largest annual portfolio in the history of the mechanism, has received a total of approximately $85 million in donations from at least 42 different donors since August 2008 (majority of the funding was triggered by the 2010 Earthquake).

The funding for ERFs decreased dramatically in 2011 (by 50 per cent) in relation to the fact that the Haiti earthquake and the Pakistan floods accounted for 75 per cent of the funding in 2010.

The $73 million contributed in 2011 was critical in meeting new developments, including the Horn of Africa crisis. Ethiopia and Kenya, in comparison with 2010, received triple the amount of funding in 2011 due to the situation. The total ERF expenditure in 2011 was $99 million. Of the total, the largest amount went to water and sanitation and nutrition sectors. NGOs received 73 per cent of the total allocations.

The following table, taken from the TORs for this assignment, shows the level of donations for the focus years of the evaluation.

Exhibit 1.1 Emergency Response Funds active during the period from 2009 to 2011

ERF Start

Dates Donations 2009 Donations 2010 Donations 2011 Total 2009 to 2011

Afghanistan 2009 $3,450,250.00 $2,829,261.00 $4,800,880.00 $11,080,391.00

Colombia 2009 $1,425,274.00 $2,130,836.00 $2,410,470.00 $5,966,580.00

DRC 2004 $0.00 $0.00 $1,000,000.00 $1,000,000.00

Ethiopia 2006 $45,631,278.00 $15,471,870.00 $42,952,693.00 $104,055,841.00

Haiti 2008 $0.00 $81,833,084.00 $471,647.00 $82,304,731.00

Indonesia 2001 $2,986,650.00 $728,200.00 $2,340,450.00 $6,055,300.00

Kenya 2009 $2,592,709.00 $0.00 $3,751,651.00 $6,344,360.00

Myanmar 2007 $1,988,271.00 $0.00 $2,199,674.00 $4,187,945.00

oPt 2007 $7,551,379.00 $5,313,378.00 $2,432,646.00 $15,297,403.00

Pakistan 2010 $0.00 $36,655,292.00 $1,951,726.00 $38,607,018.00

Yemen 2010 $0.00 $2,510,023.00 $6,499,960.00 $9,009,983.00

Zimbabwe 2008 $3,883,495.00 $717,538.00 $888,415.00 $5,489,448.00

Nepal 2008 - 2011 $0.00 $0.00 $0.00 $0.00


4

March 2013

UNIVERSALIA

ERF Start

Dates Donations 2009 Donations 2010 Donations 2011 Total 2009 to 2011

Uganda 2009 - 2011 $313,972 $618,687.00 $0.00 $932,659.00

Iraq 2007 - 2010 $621,012.00 $0.00 $0.00 $621,012.00

Total 2009 - 2011 $290,952,671.00

ERFs operate across a range of humanitarian challenges as arrayed by cluster. The following exhibit shows ERF allocations by cluster.

Exhibit 1.2 ERF Allocations by Cluster

The nature of the level of contributions to ERF also is necessary to ensure an adequate understanding of how ERFs function. The first exhibit below illustrates the steady pace of the growth of the ERF mechanism, highlighting the two most recent spikes Haiti and Pakistan.


March 2013

5

UNIVERSALIA

Exhibit 1.3 Contributions to ERF over the last 10 years

It is equally useful to understand the degree of concentration in the donorship of ERFs. The following exhibit shows a largely EU concentration, with exception of Saudi Arabia for Haiti and India for Pakistan. This issue of donor concentration will figure in the discussions with donors and the exploration of issues such as why donors choose or choose not to participate in ERFs.


6

March 2013

UNIVERSALIA

Exhibit 1.4 Donorship concentration of ERFs

Finally, the usage of ERF resources in terms of the nature of the partner organisations enables the evaluation to better target its outreach and also to test whether one of the assumptions about ERF namely its NGOs focus has been achieved.


March 2013

7

UNIVERSALIA

Exhibit 1.5 Resources Allocation by Organization type

It is important however to place the level of funding for ERFs in the context of the totality of humanitarian findings. For example, CERF in 2011 allocated some $426 million, over four times that of the ERFs. Likewise, the total amount of support for humanitarian work was in the range of $6 billion, some 66% of the estimated requirement of $9 billion.

11 .. 22 .. 22 OO uu rr II nn ii tt ii aa ll UU nn dd ee rr ss tt aa nn dd ii nn gg oo ff CC oo nn tt ee xx tt aa nn dd FF uu nn cc tt ii oo nn ii nn gg oo ff EE mm ee rr gg ee nn cc yy RR ee ss pp oo nn ss ee FF uu nn dd ss

Readers Note: The following was developed based on the review of initial documentations, including past evaluations/audits, and discussions with internal and external stakeholders.

The Rapidly Evolving Nature of ERFs

Although ERFs have been in existence for 15 years, they remain, as a funding mechanism, in a rapidly evolving state. In this Inception Report, it is not necessary to recite the history of ERFs or the findings of the 2006/2007 Evaluation of ERFs. It is important to stress, however, that OCHA has been engaged in a near continuous process of evolution of the paradigm of a country-level pooled fund, largely at the country level.

ERFs from the outset remain firmly focused at the country level, with the ERF manager and staff as the main OCHA players for their establishment, their funding and their outreach via small scale program granting largely to NGOs. The evolution of ERFs, especially since the above-


8

March 2013

UNIVERSALIA

noted evaluation report, has been one largely at the country level of an attempt to balance the recommendations of that evaluation and subsequent country evaluations and audits with the need to maintain the level of flexibility and responsiveness which is the primary characteristic as a humanitarian response mechanism, and which is recognized in all prior studies as the fundamental characteristic of the ERF itself. There needs therefore, to be a careful balance between, among a number of factors, the necessity to ensure financial and programmatic probity with mechanisms that allow for flexibility and responsiveness. It also should be emphasized that the ERFs have to exist, as do all granting mechanisms, within the overall administrative and financial paradigm of the United Nations as a whole. Thus, it is simply unrealistic to consider that one initiative like ERF could be given administrative and financial jurisdiction that would exceed anything contemplated within the totality of the UN system.

It is also important to note that OCHA itself has taken steps to begin to address this balancing act. The establishment of the FCS as a sub-unit of the CERF Secretariat in 2009 began to give OCHA the policy and programmatic coordination tool. Likewise, the establishment within the AO of a dedicated financial officer clearly responds to earlier perceptions that ERF documentation was getting lost in the midst of the totality of OCHA. However, it also should be noted that over the past three years, while progress has been reported, the challenge of developing a set of guidelines remains unfulfilled, although a current draft set is in circulation and hopefully will benefit from the findings of this current evaluation.

The development of such guidelines again has to face the tug and the pull between standardization and flexibility. Given the history of ERFs as largely country driven and in the past with degrees of administrative and procedural variances, it will be a challenge to develop a set that can respond and be responsive in instances of ERFs with annual values of less than $2 million, versus ERFs which, in the case of Pakistan or Haiti, could reach into the tens of millions of dollars per year.

In relation to this evolution, it is also important to note that key stakeholders have indicated that the success of a number of ERFs has been, in part, determined by the field level leadership in establishing and funding them.

ERFs in the Totality of Humanitarian Funding

The ERF mechanism is clearly a relatively small tool that is selectively used to provide a rapid response capacity on one hand and, as several stakeholders indicated, pragmatically to give an HC a limited pool of funds to lever, synergize and coordinate others. Notwithstanding the differences in fund sizes at a country level, the most recent OCHA Annual Report points out that CERF provides support in 2011 to 45 different countries, either inaugurating new funds such as the Philippines or sustaining and replenishing existing ones. By contrast, ERFs currently exist in only 12 environments.

The evaluation also has to take into account the interplay both at the country and strategic level of the use of Common Humanitarian Funds (CHFs). In 2005 the humanitarian reform initiative called for a new type of fund to be piloted, aCHF, with the aim of making funding more flexible and predictable in accordance with the Good Humanitarian Donorship (GHD) principles; the CHF was piloted in Sudan and the Democratic Republic of Congo (DRC) in 2006. These countries were selected due to the fact that they were and still are subject to complex humanitarian emergencies caused by ongoing conflict and the occurrence of natural hazards that often become disasters. There are now five CHFs in operation.


March 2013

9

UNIVERSALIA

The fund in Sudan is the largest having received $834.7 million between 2006 and 2011, followed by DRC which received $659.1 million. The smallest fund, in CAR, has only received $30 million since it began in 2008. Somalias CHF, which was converted from the humanitarian response fund (HRF) in 2010, has grown considerably attracting nearly five times as much funding in 2011 as in the previous year. Since South Sudan gained independence in 2011 a new CHF was established in the country. This fund split from the Sudan CHF which started in 2006 and which continues to operate. The only reported income to the South Sudan fund in 2011 was $66.2 million, which was transferred from the Sudan CHF.

CHFs provide government donors with an opportunity to pool their unearmarked contributions to a specific country, to enable timely and reliable humanitarian assistance. For those donors that do not have a strong presence in country or in-depth country knowledge, CHFs offer a platform through which they can channel their funding. For donors with in-country representation, funding any of the five funds in operation can lessen the administrative burden associated with bilateral funding and can promote coordination of humanitarian interventions

Over the last six years the proportion of money being channelled from the CHFs to UN agencies has declined from 81.1% in 2006 to 45.2% in 2011. Instead a greater share of funding has gone to both national and international organisations, with three out of the five funds disbursing more money to international NGOs (INGOs) than to UN agencies. The increase in funding to NNGOs totalling $38.6 million in 2011 can predominantly be attributed to the fund in Somalia that channelled 69.1% ($26.6 million) of its expenditure to 48 different local organisations in that year.

The question therefore arises as to what triggers the establishment of an ERF among this evolving range of vehicles. Logically, it would seem that many situations where CERF has provided support mainly to UN agencies or a CHF is in operation could benefit from an ERF mechanism so as to give OCHA a rapid response mechanism as well as a means to reach to civil society.

The issue of the winding up of an ERF leads to the contextual issue of what constitutes the emergency in ERF. While this issue is of equal consideration with respect to CERF, it is of larger importance in this evaluation due to the small-scale nature of ERFs versus CERF support and CHF support and the evaluation issues related to the focus of ERFs and cost benefits. In some instances, it appears that ERFs are being used as mechanisms to respond to emergency situations that largely go unnoticed. For example, in Colombia, one of the locales which will be reviewed by this evaluation, the ERF on an annual basis is 2/3 the size of the support provided by CERF which is categorized in the under-funded window. Situations such as this imply that the ERF provides OCHA and the donor community as a whole with a mechanism to respond to particular circumstances. Likewise, as will be shown in a subsequent exhibit, several of the current ERFs are funded only by one donor and only to a relatively small extent, thus calling into question cost benefit issues as well as issues of whether ERF should be used as a mechanism to accommodate donors who may not have their own infrastructure on site, or situations where only one donor is willing to provide country level support.


10

March 2013

UNIVERSALIA

The Functioning of ERFs

Many of the evaluation questions set out in the TORs for this assignment address issues of administrative and financial efficiency and thus speak to how ERFs function and what barriers may exist to a more efficient functioning. The illustration below, taken from a recent draft of the now approved ERF Guidelines, shows the number of steps and number of players involved in an optimal sequence from grant application to final closure of an individual project. In an Inception Report, we will not presume to analyze these stages. However, it is important to note their complexity and the degree of interdependence between OCHA and external bodies such as clusters and advisory and reference groups.


March 2013

11

UNIVERSALIA

It is also important to note that OCHA authority for ERFs is trifurcated, with policy authority existing in one division located in New York, with authority for country and regional offices located in a second division in New York, and with administrative and financial management located in a third division largely in Geneva but in the process of moving some operations to New York. While this is not an uncommon arrangement, a triangular arrangement such as this can lead to internal overlap and ambiguity. Accordingly, part of this evaluation will examine the internal dynamics within OCHA so as to ensure the reduction of any structural, policy or procedural barriers. Finally, with respect to the functioning of ERFs, it will be important to review their functioning in comparison to that of CERF. While CERF serve a high-level and largely UN agency clientele, in comparison to ERFs serving a small-scale country level clientele, there may be benefits in comparing the process maps and experiences of both to learn lessons, and possibly to develop more common procedures. Indeed, several stakeholders commented that they did not understand why ERF and CERF were not two wings of an overall pooled approach to funding. The evaluation team is aware of certain trust funds (unrestricted pooled funding) that have several programmatic wings, called tracks. The most notable of these is the World Bank Global Fund for Disaster Reduction and Recovery (WB GFDRR) which has three separate tracks but all within one administrative ambit.

Exhibit 1.6 The ERF Process

11 .. 22 .. 33 SS tt aa kk ee hh oo ll dd ee rr AA nn aa ll yy ss ii ss

The circle of stakeholders engaged for this Inception Phase was largely limited to those key individuals who could best inform the evaluation team about the context and key issues related to the evaluation and who could begin to explore emerging issues so as to give greater understanding of the key questions laid out in the TORs, which have been expanded into a traditional evaluation matrix. For the most part, the Reference Group and key OCHA managers in New York and Geneva constituted the circle of stakeholders that were contacted directly.


12

March 2013

UNIVERSALIA

This circle therefore did not represent country level interests and especially OCHAs crucial NNGO and INGO partners. A list of Stakeholder contacted can be found in Appendix I.

The stakeholder analysis that follows is comprised of two elements, the first, which stakeholders expressed to us as their concerns; and the second, the strategy that we intend to adopt to fully engage each group of stakeholders.

OCHA Senior Managers in New York and Geneva

OCHA senior managers were concerned about the strategic positioning of the ERFs as an element in the totality of OCHAs work with pooled funding, but more importantly, OCHAs broad mandate - coordinating the humanitarian response. Senior managers initially contacted expressed concerns not so much about the mechanisms of the ERF process but its relevance and where it fit as a tool to strengthen the overall United Nations role in the response to humanitarian crises. Some perceived that the ERF process had become largely supply driven and to some degree, highly reliant on donors; contrasting these perceptions with those related to CERF where there was a view that decision-making and allocation was more strategic in nature.

Senior managers will be seeking from this evaluation a strong strategic perspective and recommendations that will be designed to place a more administratively sound mechanism into the context of OCHAs overall coordinative mandate.

In terms of engagement, over the coming weeks the evaluation team will conduct workshops and an extensive series of senior level meetings. Given the precedent set by prior evaluations, we will explore how to engage the Under Secretary General herself, as it is apparent to us that the overall positioning of all varieties of pooled funds is of crucial importance to OCHAs future and a key element of the Transformative Agenda. We will also engage these stakeholders via the internal survey

OCHA Operational Managers and Staff in New York and Geneva

At the operational level, OCHA managers and staff expressed many of the same views about the overall relevance of the ERFs, however, through a somewhat different lens. At this level, the issues of relevance appear to be more directed toward how to ensure that the ERF, as a tool, gives an HC the means to more effectively deliver country level coordination. The expression money talks was used by a number of these stakeholders to emphasize their views that on a day-to-day basis and in an operational situation, having a mandate did not necessarily translate into being able to coordinate and thus having a discretionary reserve was seen as valuable. However, this view has an inherent internal problem in that the ERF mechanism is primarily designed to support NGOs in filling emergency gaps and in addressing life-threatening situations that may have been ignored or which have emerged very rapidly.

Operational level managers and staff were also concerned with the challenges of prioritization when to start an ERF and why; and with the decisions as to the longevity of an ERF, especially in certain circumstances where an ERF continues to function in what is a protracted and long-term reconstruction environment.

These stakeholders also expressed strong interest in the evaluation assessing some of the more tactical issues relative to process and timeliness. Several recent evaluations and audits have noted what was perceived to be avoidable delays and to some degree, overlap and duplication. It also should be noted that these same audits called into question OCHAs ability to effectively use national level advisory bodies so as to better reflect national needs and better integrate local considerations with respect to disadvantaged groups.


March 2013

13

UNIVERSALIA

These more operational OCHA managers and their staff will be engaged through several means. First, they will be invited to participate in the internal survey of OCHA stakeholders which will be circulated in late September. Second, they will be engaged through interviews and small group meetings. Third, their cooperation will be requested in broadening the overall data set and especially data related to the corporate functions of ERF.

Donors

Selected donors who are members of the Reference Group were contacted as part of the Inception Phase. Several noted that while their nations were generally in support of unrestricted funding, and thus were major participants in CERF, there was a feeling that unrestricted funding at the national level, the level of ERF, was somewhat different. Several noted that while CERF had a very large donor base committed to unrestricted and pooled funding, ERFs, for the most part, and with the notable exception of the replenishment of the ERF for Haiti following the earthquake, were supported by a very small number of donors, sometimes only one or two who appeared to be willing to make multi-year commitments to a particular nation. Donors also expressed a strong interest in this evaluation attempting to explore the results at all levels of ERF project support, as opposed to merely demonstrating stewardship of project outputs. Thus, there was some concern raised that the current approach to ERFs being OCHAs discretionary tool, was downplaying the importance of individual donors, and possibly placing at some risk the ability of some member states to show their national constituents the value of their humanitarian financing. Donors also expressed interest in examining the timeliness of ERF given the degree that ERF is portrayed a highly timely way to respond to emerging needs. Another donor-related concern was the extent to which OCHA reporting matched what national audit authorities required. Unlike the bi-lateral financing which most of these agencies pursue, the pooled approach of the ERF implies one form of reporting for all.

Donors also expressed some concern about the extent to which the selection of projects for ERF represented the optimal mix. Concerns were voiced about whether OCHA had the staff or capacity to reach out to national NGOs, as opposed to largely relying on well-known international partners. Finally, some donors expressed the desire especially in situations where there were only one or two donors, for OCHA to more tailor its ERF response to the humanitarian priorities of the key donors. Finally the issue of the usability and frequency of ERF reporting was raised during the Inception Phase. Some donors are seeking more information about value for money and the comparative advantage of the ERF mechanism so as to raise its profile with decision-makers.

Donors will be engaged at both the corporate level and the field level. They will be the subject of a series of interviews, with questioning going beyond mere perceptions, but seeking to secure from them actual documentary data about the relationships between OCHA at the corporate level in relation to ERFs and the donor in question. As well, at the country level donors will be contacted on an individual basis. Both corporate and country level donors will be asked to participate in the electronic survey.

INGOs and NGOs

The population of the Reference Group is highly skewed to internal stakeholders of OCHA and other UN bodies with scant representation from NGOs. This implies that one of the key challenges for this evaluation will be to reach beyond UN system stakeholders to engage a broader cross section of NGOs at both the international and national levels. That said, the initial feedback from NGOs during the Inception Phase was largely targeted toward their concerns about operational matters, largely at the country level. The importance of NGOs at the country


14

March 2013

UNIVERSALIA

level cannot be overstated. These are OCHAs operational partners with respect to ERFs and thus are probably the most significant set of country level stakeholders that we will engage.

The complexity of the application process, reporting requirements and the strictures on the ability to internally re-profile funding highlighted the extent to which some NGO stakeholders saw OCHA as not as responsive as it might be. NGO stakeholders clearly welcomed the ERF mechanism.

For the evaluation, the key issues related to outputs and processes will need to be reviewed in light of these concerns about timeliness, with the view to identifying any and all areas within OCHAs specific purview to effect streamlining and identify constraining factors outside OCHA's purview.

In terms of our engagement strategies for INGOs and NGOs, we will adopt a two-track process: corporate engagement and field level engagement.

In terms of corporate level with INGOs, we will reach out to them first by a series of telephone interviews, based on a listing provided by OCHA. Second, we will encourage these corporate INGO representatives to participate in the electronic survey for external stakeholders.

At the country level, we will drill deeper. We intend to allocate at least half of each field mission to reaching out to NGOs, INGOs or NNGOs alike, by means of small group meetings or individual interviews, So as to maximize outreach of our external survey and given the technological challenges that may face many NNGOs, we will consider administering a paper version of the external survey to NGO respondents so that we can maximize their participation.

Other UN Partners

Members of the UN system are key stakeholders both at the global and country levels. While we have yet to engage stakeholders at the country level, at the global level, these key partners of OCHA were largely interested in the issues of coordination and where ERFs fit, or could better fit within the range of the humanitarian response. There was some, albeit minor, concern that the rapidity of the ERF process could lead to overlap and duplication at the field level, especially in the early stages of an emergency. However, there was clear recognition that ERFs offered the humanitarian response as a whole with a flexible tool to rapidly engage civil society. Balancing this however, was some degree of apprehension about how long duration ERFs were to be closed and the blurring of OCHAs primary coordinative mandate with hands-on program granting.

The stakeholder engagement strategy for other UN partners is very similar to that for donors. It too will operate on a two-track basis of corporate and country level.

OCHA Personnel in the Field

As noted earlier, the stakeholder set for this Inception Phase did not include field personnel. Nevertheless it is essential to engage them throughout the entire process.

First, and as is shown below with respect to the planning for individual field missions (five), OCHA managers and personnel will be engaged via a series of individual and small group meetings. In advance, they will have been requested to submit documentation and recipient lists so that the field missions will be able to drill down beyond a simple recitation of facts and figures. Second, with respect to the OCHA ERF operations that will not be visited, staff and managers will be requested to participate in the internal electronic survey. As well, it is likely that our team will contact the staff on several occasions to conduct an interview to clarify issues that may have arisen as a result of the mandated desk review of OCHA operations.


March 2013

15

UNIVERSALIA

UN System Partner and Cluster Partner Field Personnel

The engagement of these sets of stakeholders will largely be limited to the five locales selected for field missions. Although OCHA personnel and NGO stakeholders will be engaged across the range of ERF locales, the logistics in briefing up potentially large numbers of stakeholders to participate only in the electronic survey would not be cost-beneficial. In the five selected locales, we will set aside time to engage these partners primarily via small group meetings so as to maximize time utilization. As well, they will be invited to participate in the electronic survey.


16

March 2013

UNIVERSALIA

2 . E m e r g i n g I s s u e s

Several largely strategic issues began to emerge during the Inception Phase.

While these issues are for the most part echoed in the TORs and in more detail in the evaluation questions now organized into a matrix (see below), the degree to which they were identified in the Inception Phase warrants specific reference as addressing them will have the effect of altering the focus of the evaluation, raising it to a largely strategic exercise, with tactical or operational overtones and not the obverse. However, it should be made perfectly clear that this set of issues described alone do not in themselves broaden an already ambitious evaluation. Rather, their identification has provided a more subtle context for the evaluation. Naturally the findings of prior evaluations and audits will inform the current evaluation and thus obviate duplication.

22 .. 11 .. 11 WW hh ee rr ee dd oo EE RR FF ss FF ii tt ??

Probably the most widespread concern raised relates to where ERFs fit in the totality of the humanitarian response and the difference between ERFs and other pooled funds. Institutional stakeholders very much seek an evaluation that will address the strategic positioning of country level pooled funds and especially those that have civil society, as opposed to UN organizations, as their recipients. In addition the issue of where ERFs fit needs to be examined in the context of how have the evolved since 1997 and if they have responded to changing humanitarian needs, especially over the last five years.

While the overall issue of the design of pooled funds in general is beyond the mandate of this evaluation, the TORs permits us to give strategic emphasis to this issue due to its elements related to clusters and overlap in particular. The impact of this strategic issue implies that we intend to give increased emphasis to exploring strategic considerations with key Headquarters level stakeholders and donors focusing our work in this area at the country level on the tactical and operational considerations related to day-to-day cluster activity and field level instances of potential overlap and duplication.

22 .. 11 .. 22 TT hh ee CC rr ii tt ii cc aa ll MM aa ss ss oo ff EE RR FF ss

There was broad consensus among all groups of stakeholders that there were questions of the critical mass of ERFs and that these questions existed in two dimensions: first, whether ERFs as a whole were large enough in comparison to the whole of the humanitarian response to constitute a significant critical mass that could make a difference through engagement with civil society; and whether individual project sizes were sufficiently large so as to produce results that could be sustained. This issue as a whole, therefore, probably is best addressed at the country level; with some consideration of the impact of the overall architecture of the humanitarian response at the global level, an echoing of where do ERFs fit.

22 .. 11 .. 33 TT hh ee TT rr aa nn ss ff oo rr mm aa tt ii vv ee AA gg ee nn dd aa

The IASC Transformative Agenda is the set of actions within which OCHA and the humanitarian system is collectively working to improve leadership, coordination and accountability of humanitarian response. These include: A mechanism to deploy strong, experienced senior humanitarian leadership to guide the humanitarian response; The strengthening of leadership capacities and rapid deployment of humanitarian leaders, to ensure the coordination architecture functions well; Improved strategic planning at the country level; Enhanced accountability of the Humanitarian Coordinator (HC) and members of the Humanitarian Country Team (HCT) for the achievement of collective results; and streamlined coordination mechanisms. OCHAs ERFs and the upcoming set of new Guidelines need to be placed


March 2013

17

UNIVERSALIA

strongly within the context of the Transformative Agenda, in particular where OCHA demonstrates progress made through the Transformative Agenda which also impacts OCHAs role as both a coordinator and source of funds. We recognize that the Transformative Agenda is a relatively new initiative and that the more retrospective elements of our evaluation will not be relevant to it.

However, given that this evaluation is firmly forward looking, the potential impacts of the Transformative Agenda on ERF will be explored at the Headquarters and field level both.

22 .. 11 .. 44 TT hh ee RR ee ss uu ll tt ss oo ff EE RR FF ss

Although the TORs clearly recognized the difficulties of attribution, during the Inception Phase we encountered a fairly consistent set of observations that called for this evaluation to at least begin to explore the hierarchy of results of ERFs, Donors in particular were interested in having some greater understanding of how ERF projects actually made a difference, as opposed to strictly delivering what was contracted for. As we noted in our Proposal, small-scale attribution is an extremely difficult task. Hence, we intend to use cutting-edge methodologies such as Contribution Analysis to begin to explore, especially at the country level, how projects were integrated with others and how they may have made a difference. We also will draw from the reports and evaluations of other partners and triangulate these with the testimony of NNGO and INGO recipient bodies, as well as that of OCHA staff.

22 .. 11 .. 55 TT hh ee OO vv ee rr aa ll ll FF oo cc uu ss oo ff EE RR FF AA cc tt ii vv ii tt yy

As the Inception Phase proceeded, it became apparent that a number of donors and stakeholders other than those from OCHA had some concerns about perceived mission creep with respect to the focus of OCHAs ERF activities. While nearly all were supportive of the existence of a timely and flexible fund directed toward civil society recipients, there was some concern that overlap and duplication of functions was occurring or that the ERFs, especially those of some longevity, were moving away from a paradigm based on emergency response to issues of human suffering and threats to life, toward a protracted response paradigm, blurring the line between the humanitarian response as a whole, and development. The most notable example given of this issue related to the role that ERFs may be playing in disaster responses and the promotion of resilience, areas most stakeholders took to be part of a protracted relief paradigm.

22 .. 11 .. 66 TT hh ee II mm pp ll ii cc aa tt ii oo nn ss oo ff tt hh ee CC oo uu nn tt rr yy BB aa ss ee dd OO rr ii ee nn tt aa tt ii oo nn oo ff tt hh ee EE RR FF MM ee cc hh aa nn ii ss mm

The genesis of the ERF mechanism is firmly country based, as is evident by the fact that until the promulgation of the current set of draft guidelines, there had never been a consolidated approach to the management of the ERF process. Initial research for this evaluation showed that many current ERFs have established their own guidelines that reflect local conditions to some degree. While there are numerous common elements, as would be the case in any granting program that required the use of UN financial systems, there are clearly instances of local variances. Throughout the Inception Phase of this evaluation, internal and external stakeholders emphasized the degree to which country contexts could materially alter the way ERFs were both designed or put in place in the first instance, and in the second, how ERF-supported programming was delivered. This leads to a degree of tug and pull within the evaluation itself.

Clearly, the efficiency and effectiveness of ERFs during the review period of 2009 to the present must be reviewed through a largely individual country-focused lens. However, it is equally


18

March 2013

UNIVERSALIA

important to consider how the ERF mechanism can contribute to the overall global humanitarian response. This has led us in earlier elements of this Inception Report to emphasize how ERFs may fit, both at a national and an international level.

As well, and as has been noted elsewhere, it is important to recognize the country capacity of the NNGOs in particular who partner with OCHA to deliver ERF programming. While the predominate strategic objective of ERF is to provide rapid response to unforeseen crises that provide life-saving interventions, it is very clear that from the commencement of ERF in Angola, 15 years ago, secondary objectives of the strengthening of local NGO capacity have been present.

Finally, given that the current draft guidelines have yet to be promulgated, the predominant country-based orientation of the ERFs needs to be considered in terms of performance measurement and best practices. In relation to performance measurement, given this country orientation, it is probably best to consider success in terms of how projects were defined at the country level and create within the project sample provided, a rudimentary Theory of Change which would attempt to link the largely output based project design with higher level long-term impacts. A provisional Theory of Change has been developed and is included in this Inception Report in the context of the relationship between the existing Evaluation Framework and the Evaluation Matrix which was developed for this Inception Report.

22 .. 11 .. 77 TT hh ee LL ii ff ee CC yy cc ll ee oo ff EE RR FF ss

It is apparent that the ERF process is rapidly evolving and that conditions that might have prevailed two years ago in response to the 2007 Formative Evaluation, probably have been superseded by more recent developments and even today, by the anticipated promulgation of a set of ERF guidelines and tools. Most stakeholders identified flexibility as a key characteristic of a responsive approach and were of the view that timeliness and local sensitivity were key positive characteristics of ERFs. However, few understood how ERFs came to be in the first place and how OCHA manages their life span. This again also echoes the strategic issues of where do ERFs fit and why is an ERF needed. Allied to this issue are those of how donors perceive the benefits of ERF and why some choose to participate in them. It also addresses the perception of some stakeholders that one of the benefits of an ERF is to provide medium level donors with a mechanism to funnel resources to civil society groups in instances where the donor does not have the infrastructure itself to respond to the emergency. This therefore, leads to the issue of what are the costs and benefits of accommodating donors in this manner; and also, can ERFs be used as a mechanism to encourage broader donor pools, or potentially even new donors, other than those from OECD nations.

22 .. 11 .. 88 GG ee nn dd ee rr aa nn dd DD ii ss aa dd vv aa nn tt aa gg ee dd PP ee rr ss oo nn ss CC oo nn ss ii dd ee rr aa tt ii oo nn ss

During the Inception Phase, we came to recognize that the management and administration of the ERFs had to take into account considerations of gender equality and the impact on women and girls, children in general, and the elderly.. To this end, the evaluation team will give special emphasis to how ERF related activities and procedures address the needs of females, children and other disadvantaged groups, who in many instances are impacted more seriously in humanitarian emergencies than other types of people. For greater clarity, the notion of other disadvantaged groups may, in certain circumstances under review, include ethnic or religious minorities. A specific section of the Report will address gender and disadvantaged persons considerations.


March 2013

19

UNIVERSALIA

22 .. 11 .. 99 EE vv aa ll uu aa tt ii oo nn ss mm aa kk ii nn gg aa DD ii ff ff ee rr ee nn cc ee

A number of stakeholders expressed concerns that the evaluation process in multilateral bodies sometimes did not lead to the necessary degree of organizational learning and change that had been anticipated. These stakeholders noted that across the multi-lateral system, there was a tendency to accept broad sets of recommendations; but not necessarily to move decisively toward their implementation. In essence, the lessons learned from evaluations were not being picked up to a sufficient degree. As well, and in particular with respect to this evaluation, there were calls for the evaluation to seriously focus on positive ways to move forward, and not simply to make generalized recommendations. In light of these observations, we will develop the recommendations set of the eventual Report based on a typology that has its roots in the use of Outcome Mapping as a strategic planning tool as opposed its use a tool in impact analysis. In this context, we will present a matrix of recommendations:

Those that are critical or essential to the future of country-pooled funding;

Those that are important but not essential or possibly only for selective application;

Those that would improve ERFs but only to a lesser degree; or which involve allocations of resources that might not be commensurate with their eventual contribution.

In laying out this three-level typology however, we hope that the pressures of time to refresh OCHA through the work of several internal units do not supersede the findings and the recommendations of this evaluation.


20

March 2013

UNIVERSALIA

3 . E v a l u a t i o n M e t h o d o l o g y

In our proposal, the methodology proposed for the assignment consisted of a mixture of qualitative and quantitative data collection, with attention both to forward-looking, improvement oriented questions and to assessment of impact. In summary, we will follow the OCHA Evaluation Policy1 objective of determining:

Is the right thing being done;

Is it being done well;

Could it be done any differently?

The general steps envisaged for this evaluation remain largely unchanged, and include the following steps:

Data collection (including survey administration, field missions and desk-top country cases);

Data analysis;

Presentation of initial findings at the conclusion of data collection;

Preparation and submission of a draft final report for comments by OCHA;

Preparation of the Final Report, including executive summary and all appendices; and

Presentation of the final findings to the Evaluation Reference Group.

33 .. 11 OO vv ee rr aa ll ll DD aa tt aa CC oo ll ll ee cc tt ii oo nn PP ll aa nn

33 .. 11 .. 11 FF ii ee ll dd WW oo rr kk PP ll aa nn

The Inception Phase has highlighted the importance of field missions for this evaluation. Each field mission will comprise the following steps:

Initial contact made with the selected country office for logistics;

Request for documents and initial suggestions on key informants. For the latter, both key informants currently in-country and those who have moved elsewhere will be requested;

Universalia team members will organize and carry out interviews with key respondents who are no longer in-country using telephone or Skype;

Document review;

Field mission carried out team members, including in-person interviews with key respondents and PPT-based Exit Debrief followed immediately thereafter by the Mission Report (see below).

Based on initial document review and suggestions received from our key respondents during the Inception Phase, Universalia proposes to conduct five field missions in countries affected by humanitarian natural disaster or in complex emergencies. The countries have been selected using the following criteria laid out in the matrix below:

1 OCHA Policy Instructions: Evaluations, 2010.

Glo

ba

l E

va

lua

tio

n o

f th

e E

RF

s -

In

ce

ptio

n a

nd

W

or

kp

lan

R

ep

or

t

Marc

h 2

013

21

UN

IVE

RS

ALIA

Exh

ibit

3.1

E

merg

en

cy R

esp

on

se F

un

ds S

ele

cti

on

Cri

teri

a

( )

= r

equ

ire

d b

y T

OR

s f

or

fie

ld m

issio

ns

*= s

pe

cific

ally

exc

lud

ed

by

OC

HA

fo

r fie

ld m

issio

ns

# =

clo

se

d a

nd

un

like

ly t

o b

e v

iab

le fo

r fie

ld m

issio

ns

E

RF

Date

s

Typ

e o

f E

merg

en

cy

Sin

gle

D

on

ors

2010/2

011

Mo

st

Recen

t E

valu

ati

on

/ A

ud

it

CE

RF

2011

$ m

Level o

f D

evelo

pm

en

t E

RF

To

tal $

2009 t

o 2

011

Afg

hanis

tan

2009

Com

ple

x no

0

LD

C

$11,0

80,3

91.0

0

(Colo

mbia

) 2009

Com

ple

x no

$6

LM

IC

$5,9

66,5

80.0

0

DR

C

2004

Com

ple

x ye

s

2006 E

valu

ation

$4

LD

C

$1,0

00,0

00.0

0

Eth

iopia

2006

Natu

ral dis

aste

r(s)

no

2008 E

valu

ation

$46.5

LD

C

$104,0

55,8

41.0

0

Haiti*

2008

Natu

ral dis

aste

r(s)

no

2011/1

2 A

udit a

nd

Evalu

ation

$10.5

LD

C

$82,3

04,7

31.0

0

Indonesia

2001

Natu

ral dis

aste

r(s)

yes

2010 E

valu

ation

0

LM

IC

$6,0

55,3

00.0

0

Ken

ya

2009

Com

ple

x and N

atu

ral

dis

aste

r no

2010

Evalu

ation

$22.7

L

$6,3

44,3

60.0

0

Mya

nm

ar

2007

Natu

ral dis

aste

r ye

s

$5

LD

C

$4,1

87,9

45.0

0

(oP

t)

2007

Pro

tracte

d c

om

ple

x no

$4

NA

$15,2

97,4

03.0

0

Pakis

tan*

2010

Natu

ral dis

aste

r

no

2010 E

valu

ation

$32.4

L

$38,6

07,0

18.0

0

Yem

en

2010

Com

ple

x no

$15

LD

C

$9,0

09,9

83.0

0

Zim

babw

e

2008

Natu

ral dis

aste

r no

$15

LD

C

$5,4

89,4

48.0

0

Nepal

Clo

sed #

2008 -

2011

$0.0

0

Uganda

Clo

sed #

2009 -

2011

$932,6

59.0

0

Iraq

Clo

sed #

2007 -

2010

2008 E

valu

ation

$621,0

12.0

0

$290,9

52,6

71.0

0


22 March 2013

UNIVERSALIA

The above criteria along with advice provided by various stakeholders with respect to the locales for the field missions has been used to determine the locales below. It should be emphasized that no one factor shown above was predominate. Rather, a holistic approach was used which weighed and balanced these factors along with the recommendations made by stakeholders

These factors need to be taken as a whole so that the five venues reflect a broad range of differing circumstances. In the first instance, the size of the ERF in question should be taken into consideration. Earlier, it was noted that the issue of critical mass, simply, whether a small ERF could be expected to generate sufficient results, would be a crucial issue in this evaluation. Accordingly, we took into consideration the size of ERFs in developing our proposals for the three optional venues. In examining recent OCHA Annual Reports, we noted that a number of ERFs in recent years, in essence, are the product of only one donors intervention. For the most part, these ERFs are also small scale. It is likely, therefore, that the patterns for establishment, donor relations, cluster relations, and possibly even areas of subject matter intervention may differ in this particular group of ERFs in comparison with ones with multiple donorship. The nature of the emergencies in question and the operating environment, and more particularly the impact of conflict situations, also needs to be considered when selecting the three discretionary locales. Finally, longevity should be taken into consideration; and in particular whether the ERF in question is responding to true real-time emergency needs or has evolved more toward a discretionary fund for specific protracted relief operations. In addition, we have taken into account the instances where the CERF Underfunded window has been used, for example, in the case of Colombia.

It is also necessary to emphasize that this overall evaluation of ERFs is not simply a compilation of case studies. It was designed as a synthesis, where varying lessons from different cases can be brought together to provide a more strategic assessment of the role of ERFs, how and why they have evolved as they have, and how ERFs or similar mechanisms can fill perceived gaps. This is to say therefore, that case study reports derived from these five field missions will not be judgemental in the sense of a technical audit. Rather, they will look toward problem solving and mechanisms to maximize their competitive advantage.

Given these consideration, Universalia presents below a summary table of the countries proposed for field missions. The Exhibit below categorizes the five field missions.

As noted above, two are mandated. Both, in our estimation, are excellent candidates due to the diversity and complexity of their situation, let alone that they have not been subject to a recent evaluation or audit. One, Ethiopia, reflects a near unanimous opinion that while Ethiopia has been the subject of many reviews since 2007, its shear size and diversity requires that it be considered as part of this exercise.

The last two reflect the capability of providing some options.

A choice between Afghanistan or Yemen (which attracted considerable stakeholder support) reflects the need to address highly complex and volatile emergency situations; with the Yemen case being be more indicative of the traditional model of internal conflict, as opposed to Afghanistan where the presence of international forces may complicate the humanitarian response. Finally, the choice between Myanmar and Indonesia reflect considerations relative to ERFs that are operating to some degree in isolation and in particular with only the support of one donor and only at relatively modest levels. In these cases, crucial issues of economy of scale and cost benefit come into play as well as those related to starting or concluding an ERF.


March 2013

23 UNIVERSALIA

Exhibit 3.2 Tentative Field Mission Plan and Options

Country Estimated Time in the

field

Tentative dates of mission

September 20- November 1

early/ middle/ late

Mandated

Colombia 8 days early

oPt 8 days early

Broad / To Unanimous Consensus

Ethiopia 8 days middle

Options

Afghanistan or Yemen 8 days middle

Indonesia or Myanmar 8 days late

During the review of the Draft Inception Report, the evaluation Reference Group recommended the selection of Afghanistan and Indonesia. These choices were made largely in relation to security and logistical considerations on one hand, combined with the probability of being able to reach out to in-country stakeholders, both NGOs and international bodies on the other hand. The schedule set down above has subsequently been confirmed in a series of bilateral discussions. In short, the Palestine and oPt field missions will commence on Monday, October 1st. The final mission to Indonesia will conclude on November 9th. After each mission there will be time for follow-up telephone conversations so as to broaden the outreach of the evaluation.

33 .. 22 FF ii ee ll dd MM ii ss ss ii oo nn DD aa tt aa CC oo ll ll ee cc tt ii oo nn

The general approach for the field missions is one where data collection occurs throughout an entire week with the debrief occurring the start of the following week. Therefore, the evaluation team will prepare the PowerPoint debrief during the intervening weekend days.

Based on experience, the following table gives a general idea of how to animate a field mission such as the ones required for this assignment. It is essential to underscore that Country Offices are expected to provide the required data well in advance, at least two weeks if not more, and that similarly, they will be expected to provide a draft schedule based on the table below at least a week in advance. Given our collective experience in conducting field missions, these deadlines serve not only to mobilize country level staff but also to give an indication to the evaluation team in advance of the likelihood of success and also provide a final window of opportunity, with OCHA Headquarters personnel to institute remedial action.

Turning to the actual stakeholders, given the information provided to the evaluation team as part of the Inception Phase, we wish to firmly underscore the need for these five Country Offices to mobilize the NNGO and INGO stakeholders giving special reference to the NNGOs.

Exhibit 3.3 Preliminary schedule for field missions

Day Event

1 Briefing with ERF Manager

1 Briefing with Humanitarian Coordinator

1 Focus group with UN IPs

1/2 Focus groups/Meetings/Interview with Advisory/Review Board(s) members


24 March 2013

UNIVERSALIA

Day Event

2/3/4 Focus groups/interviews with INGO IPs

2/3/4 Focus groups/interviews with NNGO IPs

2/3/4 Focus group with cluster coordinators

4/5 Meeting with Government emergency response entity

4/5 Meetings/interviews with donors

4/5 Follow-up interviews with UN, INGO and NNGO IPs

5 Clarification meeting with ERF Manager (and staff)

6/7 Prepare exit PPT debrief

8 Exit debrief/ Validation workshop

14 Submit aide memoire

20 Receive comments

24 Distribute final aide memoire

33 .. 33 SS pp ee cc ii ff ii cc MM ee tt hh oo dd oo ll oo gg ii ee ss

33 .. 33 .. 11 DD oo cc uu mm ee nn tt rr ee vv ii ee ww

The central document review will cover key internal documents provided by OCHA, both at the Headquarters as well as the Country Office levels, including strategy, planning and annual reporting documents linked to ERF activity. These should include information extracted from OCHAs Financial Tracking System (FTS). The Evaluation Team will also review other key programming documents relating to the design and management of pooled funds, such as specific policies and guidance notes. We will review additional evaluations and studies linked to pooled funding and humanitarian response, as well as those related to the Cluster response to humanitarian disasters.

As was noted in the early segments of this Inception Report, there are problems with the scope and quality of the data set that OCHA Evaluation and Guidance was able to amass prior to the commencement of this assignment. The evaluation team has provided OCHA with a list of additional materials that will be essential; and in the Evaluability section of this report comments on not only what is required but why it is required.

33 .. 33 .. 22 DD ee ss kk RR ee vv ii ee ww ss

The revised TORs for this assignment specified that the evaluation will conduct a desk review of 15 ERFs, including three that have been closed or that are in the final stages of operation. To ensure consistency of analysis, an ERF desk review template has been developed (Appendix X).

The desk reviews are intended to add breadth and scope to our evaluation by going beyond what can be achieved through field missions. Each of them will include the following steps:

Document review based on the data set provided by OCHA Evaluation and Guidance;

Telephone/Skype interviews with 3-5 key informants.

In terms of timing, and because the desk reviews also include the five locales selected for field missions, these reviews provide probably the most solid basis on which to assess the ERFs in general and in particular, provide the solid basis for field missions which will go beyond merely


March 2013

25 UNIVERSALIA

collecting data about what happened and concentrate on why it happened and what are its consequences. To that end, we plan to commence these desk reviews as soon as possible after the acceptance of the Inception Report.

Again, it must be emphasized that the utility of these desk reviews is completely dependent on the adequacy of the data set.

33 .. 44 AA dd dd ii tt ii oo nn aa ll ii nn -- pp ee rr ss oo nn II nn tt ee rr vv ii ee ww ss ,, TT ee ll ee pp hh oo nn ee II nn tt ee rr vv ii ee ww ss // FF oo cc uu ss GG rr oo uu pp ss

In addition to interviews held during the five field visits and as a part of the fifteen desk reviews, interviews will be carried out in with key informants across the globe. They will be guided by semi-structured interview protocols. The following stakeholder groups will be interviewed, both in person and through telephone interviews.

OCHA HQ staff in Geneva and NY;

Interviews and focus group with the donors group(s) in Geneva and/New York and also at their HQ locations;

Interviews and focus groups with members of major international NGOs;

Interviews with members of IASC Cluster Groups and Working Groups other members of the UN family with strong links to pooled funding, such as the World Food Programme, UNICEF.

33 .. 55 AA dd dd rr ee ss ss ii nn gg UU nn ff oo rr ee ss ee ee nn CC ii rr cc uu mm ss tt aa nn cc ee ss

Like any evaluation unforeseen circumstances can influence its pace and direction. For this process we have chosen to work in a very proactive manner, securing as much field level data in advance and thus already having identified some unanticipated limitations, for example, the magnitude of project files thus necessitating a sampling technique. As well advance access to field level material has enabled the team to better focus the short duration field missions, again for example, toward NNGO respondents. The pace of the evaluation enables us to have a period of virtual follow up at the country level before the submission of the country case study.

33 .. 66 SS uu rr vv ee yy ss

Universalias original Proposal did not include electronic surveys. However, as part of the Inception Phase, it became evident that surveys would be required to bolster the scope of the evaluation and especially to reach stakeholders in those locales which have not been selected for field missions.

We now intend to conduct two electronic surveys distributed either by OCHA Evaluation and Guidance or ourselves.

One will be directed toward institutional stakeholders at the global and country level (donors, HQ level personnel of UN agencies, country level personnel of UN agencies, OCHA staff at both Headquarters and in the field the ERF locales). The other will be directed towards external stakeholders and will focus largely on INGOs and NNGOs in all the present operational ERF environments. As well, Headquarters personnel from INGOs will be engaged.

The surveys, drafts of which are included as Appendices II and III, focus on perceptions of the benefits of ERFs and any issues that detract from these overall benefits.

We estimate that it would take about 20 minutes to complete the survey.


26 March 2013

UNIVERSALIA

In terms of timing, we propose to electronically distribute the surveys in the last week of September with a follow-up reminder two weeks later. This timing has been selected to give OCHA Evaluation and Guidance and OCHA Country Offices in the ERF locales the maximum time to amass the electronic coordinates of both the internal and external stakeholders. In certain circumstances, as yet to be determined, the field mission team may choose to administer the survey on-site in a paper format so as to fill any gaps that may be perceived to exist in the information provided by Country Offices.

33 .. 77 RR ee vv ii ss ee dd EE vv aa ll uu aa tt ii oo nn MM aa tt rr ii xx

The Evaluation Matrix is based on the primary evaluation questions listed in the Terms of Reference which are derived from the ERF Results Framework (below). The matrix amplifies on each key question by articulating sub questions, illustrative indicators, and data sources.

Exhibit 3.4 ERF Results Framework

33 .. 77 .. 11 AA PP rr oo vv ii ss ii oo nn aa ll TT hh ee oo rr yy oo ff CC hh aa nn gg ee

The above Results Framework which was developed by a consortium of actors several years ago, adequately describes the expectations which were laid down at that time and in the accompanying documentation laid out a set of relevant indicators for the key elements noted above. However, this Results Framework, given its nature as a mechanism to lay out performance assessments, may not adequately capture the hierarchy of anticipated outcomes


March 2013

27 UNIVERSALIA

and the linkages between outcomes, risks, external factors and some of the program logic assumptions that underpin the articulation of these outcomes.

Accordingly, so as to better visualize the ERF process, the above Results Framework was adapted into a Theory of Change model which is laid out below.

In this model, the various levels are identified by the stars and the accompanying letter.

Each star reflects either a part of the existing Framework or assumptions, etc., which were articulated in the Frameworks accompanying documentation.

Level A reflects what are generally seen to be inputs. The accompanying assumptions and risks transpose the lowest line of the Results Framework into Theory of Change elements. It is important to note that for each successive hierarchical level, this provisional Theory of Change did not develop any additional performance indicators other than those which were identified in the CHF Framework Paper.

Level B articulates what was described in the Framework as Initial and Bridge Funding Provided.

Level C expands upon the Framework by adding in a set of short-term outcomes which are directly related to the three elements of the Framework.

Level D rearticulates the three medium-term outcomes laid out in the Framework. Their achievement is amplified in the accompanying set of assumptions, risks, and indicators which were drawn from the 2nd to top line of the Framework.

Level E directly parallels the highest level of the Framework and has been articulated as a long-term outcome with accompanying assumptions and risks.

In short, the provisional Theory of Change below has re-positioned the elements of the existing Framework. It does not supplant the existing Framework. Rather, it is designed to accompany it and to enable the Evaluation to better assess the logical assumptions and levels of risk which may be inherent in the attainment of the hierarchy of Results which was previously articulated.


28 March 2013

UNIVERSALIA

In some cases, the questions contained in the Evaluation Matrix have been modified from those in the Terms of Reference, in order to reflect additional information as collected during the Inception Phase.

Several areas in particular were consolidated as Evaluation Questions laid out in the TORs were duplicative or in fact, sub-questions of a more over-arching key question. It should be noted that the TORs itself contained a large number of Evaluation Questions which necessitated the development of sub-questions so as to amplify their intent.

Two areas were expanded upon and broken out as specific Evaluation Themes. These were Partnerships and Performance Management.

In both instances, the evaluation team felt that the sole Evaluation Question for both these areas insufficiently addressed their importance, especially in light of the degree to which issues related to these two areas figured largely in the 2007 Formative Evaluation of the ERFs and in several of the subsequent stand-alone evaluations of individual ERFs.

Several areas of inquiry may be somewhat problematic. While the evaluation does not specifically ask for the assessment of beneficiary impact, it is implied throughout and was seen by some stakeholders as an important element. For example, the ERF as a concept was designed to respond to rapidly emerging life-threatening incidents. It is therefore inherent that this more substantive evaluation begin to address whether in fact, in such circumstances, lives were actually saved. However, given the fact, as noted earlier, that OCHAs ERFs constitute only a small fraction of most humanitarian responses, it would be prohibitively difficult to attempt to identify specific impacts. To address this problem of the assessment of the contributions that


March 2013

29 UNIVERSALIA

ERFs make to longer term outcomes, we will rely on a variety of methodologies. First, we will work with country level staff to develop a rough theory of change model at the project level in the context of Contribution Analysis which will begin to guide us toward going beyond output measurement. Second, we will seek the cooperation at both Headquarters and country level, of OCHAs institutional partners and mainly the recipients of CERF funding to secure from them any relevant reports, audits or evaluations that have been conducted and which might relate to the kind of activity where ERF was involved. Finally, we will also seek out the testimony of OCHA staff and INGO and NNGO recipients of ERF with respect to the results of the ERF support, and equally, in light of Contribution Analysis, the interplay between various interventions that may have collectively contributed to the attainment of broader goals. For greater certainty, we will not engage individual beneficiaries directly due to time and logistics limitations.

Glo

ba

l E

va

lua

tio

n o

f th

e E

RF

s -

In

ce

ptio

n a

nd

W

or

kp

lan

R

ep

or

t

30

Marc

h 2

013

U

NIV

ER

SA

LIA

Exh

ibit

3.5

E

valu

ati

on

Matr

ix

Issu

e/T

hem

e

Majo

r E

valu

ati

on

Q

uesti

on

s/S

ub

-Th

em

e

Exam

ple

of

Su

b-Q

uesti

on

s

Illu

str

ati

ve In

dic

ato

rs2

So

urc

es o

f D

ata

Op

era

tio

nal Im

pact

Cri

tical u

nfo

reseen

h

um

an

itari

an

need

s

ad

dre

ssed

Ho

w, and to w

hat exte

nt

has E

RF

mechanis

m

contr

ibute

d to the

hum

anitarian c

om

munity

s

abili

ty to a

ddre

ss c

ritical

unfo

reseen h

um

anitarian

needs in a

tim

ely

an

d

eff

ective m

anner?

What specific

featu

res o

f th

e

ER

F lend them

selv

es to

impro

vin

g the h

um

anitarian

response?

What are

the v

iew

s o

f re

cip

ients

and n

ational govern

ments

re

gard

ing the o

vera

ll opera

tional im

pact of

the

ER

Fs?

What are

the v

iew

s o

f oth

er

develo

pm

ent part

ners

donors

oth

er

UN

etc

.)?

Evid

ence o

f new

featu

res/n

ew

typ

es o

f pro

jects

/recip

ients

/gaps f

illed

Evid

ence o

f le

vera

ge o

f E

RF

with o

ther

hum

anitarian d

onors

Response r

ate

s f

or

ER

Fs

sin

ce 2

007

(tim

elin

ess)

Rate

of

satisf

action w

ith E

RF

by

national

govern

ments

Rate

of

satisf

action o

f re

cipie

nts

Rate

of

satisf

action w

ith o

ther

UN

part

ners

Rate

of

satisf

action w

ith d

onors

Identification o

f E

RF

uniq

ue f

eatu

res a

nd

degre

e o

f satisf

action in/ accepta

nce o

f

them

Degre

e o

f satisf

action in the tim

elin

ess o

f E

RF

s

Inte

rnal and e

xte

rnal

surv

eys

Desk R

evie

ws

Countr

y docum

ent

revie

ws

Report

s / a

udits

HQ

docum

ent

revie

ws

Inte

rvie

ws w

ith

inte

rnal and e

xte

rnal

sta

kehold

ers

Sm

all

gro

up

meetings

Fin

ancia

l /

transaction d

ata

What has the p

erf

orm

ance

and valu

e a

dded of

the

ER

F m

echanis

m b

een in

the c

onte

xt of

the o

vera

ll hum

anitarian a

rchitectu

re

and h

um

anitarian r

efo

rm

pro

cess (

e.g

. R

C/H

C

str

ength

enin

g, clu

ste

r appro

ach, develo

pm

ent of

eff

ective p

art

ners

hip

betw

een U

N a

nd n

on-U

N

hum

anitarian a

cto

rs,

hum

anitarian fin

ancin

g)?

Ho

w h

as the R

C/H

C f

unction

been a

ffecte

d?

Is there

evid

ence o

f in

cre

ased

clu

ste

r coopera

tion?

What are

the s

pecific

valu

e

added

ele

ments

?

Has the E

RF

pro

duced

syn

erg

ies o

r cata

lyzed

additio

nal fu

ndin

g?

Rate

of

satisf

action in E

RF

model by

clu

ste

r part

ners

Identification o

f changes in R

C/H

C r

ole

s

Changes in o

vera

ll hum

anitarian f

i

Date post:	26-May-2018
Category:	Documents
Upload:	lamdang
View:	212 times
Download:	0 times