CSH/UNHCR/20 0106 - ToR Evaluation - UNHCR Saving maternal newborn lives (Final).docx 1/15

Evaluation of the project “Saving Maternal and Newborn lives in Refugee Situations” in

Cameroon, Chad and Niger

Key Information at glance about the evaluation

Title of the evaluation: Evaluation of the project “Saving Maternal and Newborn lives in

Refugee Situations” in Cameroon, Chad and Niger

Timeframe covered: 2018-2019

Type of exercise: Project Evaluation (Decentralised Evaluation)

Evaluation commissioned by: UNHCR Public Health Section

Evaluation manager’s contact: schulteh@unhcr.org

Date 29 November 2019

1. Introduction

In January 2016 UNHCR started to implement a two-year project ‘Saving Newborn Lives in Refugee

Settings’, aiming to improve neonatal care services with a focus on low cost, high impact newborn

interventions in refugee camps in South Sudan, Kenya and Jordan. The project received support from

the Bill & Melinda Gates Foundation. An evaluation was carried out in 2018, covering the period 1

January 2016 to 31 October 2017. Prior to this evaluation, positive effects had already been reported

from the health providers, partners and MoH authorities as well as from UNHCR teams in the field who

indicated that the project significantly contributed towards improving the quality and availability of health

care services to mothers and newborns in the refugee settlements. UNHCR proceeded at an early stage

with a request for the expansion of the project to three further countries hosting refugees and adding

maternal health and family planning to the newborn approach. The project expansion was approved

and received additional support from the Bill & Melinda Gates Foundation.

Building on the original ‘Saving Newborn Lives in Refugee Settings’ project, the expansion of the

project in 2018-2019 to Cameroon, Chad and Niger aimed to benefit from the consolidation of learnings

and practice and an extension of action to address targeted maternal, newborn and

contraception/family planning care interventions known to save maternal and newborn lives.

The implementation of the expanded project ends 31 December 2019.

With the end of project evaluation UNHCR aims to assess achievements, enhance learning and to

help identify contextual and organizational factors that may have had a particular enabling role in the

project development and those that may have slowed progress. Further, the evaluation is meant to

investigate the sustainability of the achievements in the current configuration of actors and funding and

identify the essential support necessary to maintain achievement, potentially expand coverage and/or

to complement any element identified as missing or insufficient. Finally, the evaluation will include some

costing analysis of the key components of the project per country.

2

2. Subject of the evaluation and its context

UNHCR aims to ensure that all refugees can exercise their rights to access essential public health

services at the community, primary and secondary health care level.

The different settings of UNHCR’s operations pose challenges due to the wide variety of health care

systems, financing of health care and disease patterns and burdens, between regions, countries and

even sub-regions within a country. Security and access to refugee populations can present an

important additional challenge in terms of staff deployment, supervision and support visits, supply of

essential medicines and medical equipment and the referral of patients. In some contexts, the

national health care system is unable to address the national populations needs, and further struggles

with the additional burden of refugees.

Women, girls and children are disproportionately affected in conflict and emergency situations. Globally,

one in seven women will face a complication during pregnancy or childbirth. Every year, an estimated

295 000 women die during and following pregnancy1; 94 percent of all maternal deaths occur in low

and lower middle-income countries. Humanitarian settings present a particular concern: the lifetime

risk of dying in pregnancy or childbirth for women in fragile settings (countries experiencing crisis or

conflict) is estimated at 1 in 54 compared 1 in 5 400 in high income countries. Maternal deaths are

defined as death occurring during pregnancy and childbirth and up to 42 days following the end of

pregnancy. Direct causes of maternal mortality are those where the death is directly related to the

pregnancy, related complications or interventions; the majority of maternal deaths are due to direct

causes, most of which can be addressed with relatively simple means or prevented entirely. A recent

analysis conducted by UNHCR of 72 audits of maternal deaths that occurred in 2018 in 29 refugee

camps in seven Eastern African countries found a significant contribution of the third delay to maternal

deaths and which highlights capacity gaps in provision of emergency obstetric care amongst NGO

providers and national services. The analysis concludes that while evidence-based guidance on

provision of quality Emergency Obstetric and Neonatal Care (EmONC) is available, implementation is

often far behind.

2.9 million newborns die in the first four weeks of life, and 2.6 million more are stillborn, dying in-

utero during the last three months of pregnancy. Neonatal deaths, defined as any death that occurs in

the first 28 days of life, currently account for nearly 50 per cent of all deaths of children under five years

of age in low- and middle- income countries. Approximately three-quarters of these deaths are early

neonatal deaths that occur during the first week of life, including 36 per cent that occur within the first

24 hours after birth. More than half of all neonatal deaths occur in countries with a newborn mortality

rate of 30 or more deaths per 1,000 live births. Many of these countries have experienced recent conflict

or humanitarian emergencies and are hosting refugees. The three major causes of neonatal mortality

are complications of preterm birth and low birth weight, infections, and complications that arise during

the birth process (previously known as birth asphyxia). These causes account for more than 85 per cent

of newborn mortality. Causes of newborn deaths in refugee settings are not different from causes of

newborn death globally. However, underlying risk factors and conditions that contribute to newborn

deaths can be exacerbated in refugee situations with inadequate shelter, poor sanitation and hygiene,

poor maternal diets, limited access to skilled attendance at delivery, limited capacity for care in the first

24-48 hours after delivery, and low prevalence of early initiation and exclusive breastfeeding.

1 WHO maternal mortality – Key Facts Sep 2019. https://www.who.int/news-room/fact-sheets/detail/maternal-mortality

CSH/UNHCR/20 0106 - ToR Evaluation - UNHCR Saving maternal newborn lives (Final).docx 3/15

Access to contraception and family planning services is a human right and is an essential service

in refugee operations. Scaling up access to quality contraceptive and family planning services can

reduce maternal and neonatal morbidity and mortality, prevent unwanted pregnancies, reduce rates of

abortion and adolescent pregnancy and avert related risks (including unsafe abortion). Family planning

also provides numerous societal benefits including supporting a sustainable environment, poverty

reduction, better nutrition, improving girls’ education and empowerment, and reducing HIV

transmission, among others. Women in humanitarian crisis situations, including refugee settings are

particularly vulnerable and their rights to access reproductive health services, including family planning,

must be ensured. It is estimated that 225 million women of reproductive age in the developing world

have an ‘unmet need’ for contraception – they would like to delay or stop childbearing but are not using

any form of contraception. Past reviews of family planning services2 have found a number of barriers

or gaps in family planning services in refugee operations. These include among others: inadequate and

insufficient provision of modern contraceptives methods and low awareness about these methods; low

uptake of contraception related to accessibility issues (distance and costs), lack of awareness,

opposition to use, fertility-related reasons, and religious reasons; poor service quality, including

disrespectful attitude by health providers, long waits, lack of privacy and confidentiality and poor hygiene

in facilities. Provider-level barriers include poor training; lack of knowledge; biases; and hesitancy of

health staff to discuss or offer contraception due to perceived sociocultural resistance3

Understanding the reasons why new-borns die, understanding the factors affecting the coverage and

quality of antenatal, intrapartum and postnatal care as well as those affecting access to and acceptance

of contraceptive/family planning services are crucial for improving maternal and newborn health

programming in refugee settings.

The project developed in Cameroon, Chad and Niger emphasizes the expansion of key low-cost,

high-impact interventions to address maternal together with newborn health, including use of the

partograph, active management of third stage labor, proper cord care, thermal care, initiation of

breathing and resuscitation, early initiation of exclusive breastfeeding, kangaroo mother care and eye

care as well improving access to and acceptability of contraception/family planning in refugee

settings and other components of EmONC in three selected countries.

The central activities carried through the project in 2018 and 2019 present as follows:

1. Baseline assessment, detailed design of project, monitoring, programme evaluation, learning

and dissemination – Understanding context-specific needs is the first step in improving health

programming. Because refugee operations differ considerably in the robustness and reach of health

services, programmes must be tailored to the specific opportunities and constraints of the setting to

be effective. A baseline assessment was carried out by UNHCR in the three countries to understand

context-specific needs and factors which fed into the country specific project design. The information

and recommendations provided by the baseline assessment were used to plan the most appropriate

interventions to improve services and reach the outcomes. Building on the past project in South

Sudan, Kenya and Jordan, a systematic documentation of experience and monitoring tools was

started from the beginning of the project to enable regular follow up of the progress and learning

from the project implementation in the target countries.

2 http://www.conflictandhealth.com/content/9/S1/S3%0ARESEARCH 3 http://www.unhcr.org/4ee6142a9.pdf

4

2. Capacity building of human resources – The project invested into capacity building to promote

access to quality emergency obstetric care, essential newborn interventions (including KMC),

antenatal and postnatal care; promotion of early initiation of exclusive breastfeeding for all newborns

and contraception/family planning services. Training improved professional competencies of skilled

health providers such as doctors, nurses and midwives, targeted the provision of the seven basic

EmONC signal functions, and respectful maternity care. Training followed a cascade approach using

training of trainers from each project site and used the low-dose, high frequency methodology with

the core subjects from the Helping Mothers Survive and Helping Babies Survive packages. Specific

training was developed to increase knowledge and understanding of lay-persons (home visitors,

community health workers, traditional birth attendants, ancillary staff) and to strengthen their role in

conducting pregnancy and postnatal home visits, community mobilization and improving access to

services.

3. Strengthen health facility readiness and quality. The project ensured the overall capacity of the

health facilities to provide essential maternal and newborn services and family planning services by

putting into place, renovating and maintaining the essential component of functional health services:

essential amenities, essential equipment, standard precautions, laboratory tests, and medicines and

commodities. The readiness and quality were ensured through: a) availability of skilled health

providers through training, on-site supervision and mentoring; b) ensuring guidelines in place for

maternal, newborn care and contraception services; c) providing adequate commodities, supplies

(including Kangaroo wraps, caps and towels as well as basic materials for CHWs) and equipment

for health clinics to implement clinical protocols and provide a range of modern contraceptive

methods; d) engaging communities and promoting pre-conception, contraception, delivery and post-

natal care; e) promote quality assurance by utilizing tested balanced score cards, and regular

supervisory visits; f) reinforcing technical support system through regular structured support visits

and meetings, measuring progress and identifying gaps; g) strengthening the referral systems

4. Comprehensive family planning - Contraception prevalence rate is very low in the three countries.

To increase the use of family planning methods the project addressed the following: 1) overcoming

stigma and promoting family planning through strengthening sensitization at community level; 2)

ensuring availability of a full range of modern contraceptive methods; 3) ensuring clinical guidelines

and appropriate counselling materials in all health facilities; 4) reinforcing training on contraceptive

methods and counselling to health providers (doctors, nurses and midwives) as well as training of

CHWs and TBAs on basic family planning concepts; 5) reaching vulnerable populations, including

adolescents through the provision of context specific services.

5. Community-based programmes - In addition to standard facility-based maternity and neonatal

care, community-based programs were considered. Community-based programs that are well

integrated into primary health services can help ensure a continuum of care and provide linkages to

facility-based services. Community-based interventional care packages have been found to reduce

maternal morbidity by 25%, stillbirths by 16%, perinatal mortality by 20%, and neonatal mortality by

24%. They also play a key role in the behaviour change of their communities and also increasing

referrals to health facilities for maternal complications by 40% and improved the rates of early

breastfeeding by 94%. Community-based approaches that were considered in the different projects

ïnclude: 1) building or expanding community-based support groups on basic RH issues, such as

women’s groups, Mother’s Groups, or breastfeeding support groups, youth groups, which can create

a network of support for RH and related concerns; 2) training CHWs and traditional birth attendants

CSH/UNHCR/20 0106 - ToR Evaluation - UNHCR Saving maternal newborn lives (Final).docx 5/15

on evidenced based activities regarding maternal and newborn care (breastfeeding, danger signs,

cord care), family planning, promoting ANC and PNC visits to health facilities, and strengthening

community-based referrals for mother and newborns; 3) Implementing a program of pregnancy and

postnatal home visits, which in addition to training, included implementing a fixed schedule and

content of visits, supervision plan, register and communication with health facility, among other

program supports 3) increasing IEC and sensitization programs on a community level to promote

use of family planning.

6. Expanding learnings to an organizational level - Through the two phases of the project (2016-

2018 and 2018-2020), common gaps in service delivery and management have been noted across

countries and operations. In order to disseminate learnings and integrate innovative practices into

other UNHCR operations, a Communication and Dissemination matrix was developed in 2019.

Activities include the development of guidance materials targeted at the managerial level in order to

build capacity in managing RH/MNH, such as: development of operational guidelines; webinars for

management level UNHCR staff (Public Health Officers and managers of NGO partners) on

improving newborn (n=2) and maternal (n=2) health and family planning (n=2) in refugee settings;

and the development of short High Impact Practice primers to aid implementation of key

interventions (e.g. KMC, neonatal resuscitation). Other activities include: a field support mission to

additional operations (Bangladesh) to support RH/MNH service development based on lessons

learned from BMGF project implementation; the addition of essential contraceptive, maternal and

neonatal medications and equipment to UNHCR’s Essential Medicines List (e.g. kangaroo wraps,

feeding cups, tranexamic acid, DMPA-SC, etc) based on gaps identified through the projects.

External communication activities included: dissemination of baseline assessment findings;

participation in/presentation at IAWG Newborn Health in Humanitarian settings experts meeting, and

ongoing participation in development of 5 year strategy roadmap; presentations of project lessons

learned at IAWG global meeting 2020; participation in a Spotlight session at the Global Refugee

Forum highlighting maternal and neonatal deaths and, in particular, the project in Chad; publication

of 2 peer-reviewed qualitative articles from the first phase of project; and re-development of

UNHCR’s public health website in order to make support materials more easily available to country

operations.

7. Reproductive, maternal and neonatal health population-based survey – A population-based

household survey on key RH, maternal and newborn indicators was carried out in Chad and

Cameroon in 2018/2019. In addition to the primary findings, UNHCR is coordinating with experts to

further analyse the survey data - US Centre for Disease Control for re-confirmation of primary data

analysis and University of Washington for secondary data analysis on any correlates of low birth

weight.

3. Purpose and objectives

The main purpose of this evaluation is to assess the relevance, impact and effectiveness of the “Saving

maternal and newborn lives in refugee situations” project in the three targeted refugee operations. The

evaluation should help to identify contextual and organizational factors that may have had a particular

enabling role in project implementation and those that may have slowed progress. The evaluation is

meant to assess the sustainability of the achievements in the current configuration of actors and funding

6

and identify the essential support necessary to maintain achievements, potentially expand coverage

and/or to complement any element identified as missing or insufficient. Finally, the evaluation will

include some costing analysis of the key components of the project per country.

The evaluation will be used both for learning and accountability; findings will be used to guide

programme practices to improve maternal, newborn and contraception/family planning care in refugee

operations; and to demonstrate what worked well, why, and lessons learned from implementation to the

funders and organizational leadership.

Specifically, the evaluation seeks to address the following and provide specific, actionable and

practical recommendations for future programming:

- Evaluate the extent to which project objectives and proposed outcomes were achieved by

measuring performance against each performance outcome indicator under each result area.

Analyse key determinants that, positively or negatively, influenced the achievement (or not) of these

results.

- Comparing different project countries and project sites, identify enabling factors and factors that

may have slowed progress.

- Provide recommendations on future project design including how to ensure planning, management,

monitoring and evaluation frameworks are more effective and taking account of above factors.

- Evaluate the effectiveness and efficiency of the organisational set‐up for the project, tools and

systems used in the delivery and monitoring of the project and to what extent these contributed to

delivery of the project outcomes.

- Assess the sustainability of the individual project components, and identify critical factors that may

affect sustainability and recommend support necessary to maintain achievements, potentially

expand coverage and/or to complement any element identified as missing or insufficient

- The draft report will be shared with partners in the respective project countries and discussed at

the coordination meetings to ensure ownership.

A final report will be prepared and shared with all UNHCR public health staff globally. The same

final report will also be shared with health partners in respective countries by the public health

officers and made available on the public health website.

4. Evaluation Approach

4.1 Scope

The evaluation scope – relating to population, timeframe and locations– is as follows:

Timeframe to be covered in the evaluation: April 2018 – 31th December 2019.

Population location and details:

Cameroon

CSH/UNHCR/20 0106 - ToR Evaluation - UNHCR Saving maternal newborn lives (Final).docx 7/15

- In the Extreme North of Cameroon, 57,000 Nigerian refugees live in Minawao refugee camp,

with 30,000 others living in sites outside the camp. Minawao camp has 2 health centres, one

of which provides delivery services. The health partner is International Medical Corps (IMC).

- In the East of Cameroon, UNHCR is supporting more than 185,000 refugees from the Central

African Republic in 38 health centres and seven health districts. For this project five locations

were chosen, including four refugee sites and one host village (Gbiti) where many refugees are

living together with the host population on the border with the Central African Republic (RCA).

In addition, three district hospitals that serve as the main secondary level of care were included:

DH Batouri, DH Kette and DH Garoua-Boulaï. The health partner in the East region is African

Humanitarian Action (AHA).

Chad

- In the South of Chad, 98,645 refugees fleeing war in the Central African Republic are living in

multiple districts along the border. This project includes six refugee camps and their five

primary health centres, supported by three district hospitals. The health partners in the South

are Association pour Developpement Economique et Sociale (ADES) and CSSI.

- In the East of Chad over 332,048 Sudanese refugees fleeing war in the Darfur region have

occupied 12 camps for more than a decade. This project focuses on five camps and their

primary health centres, supported by two district hospitals that serve as the secondary referral

hospitals. The health partner in the East is the International Rescue Committee (IRC)

Niger

- In the south of Niger, more than 100,000 Nigerian refugees have fled Boko Haram, and are

living in settlements along the border, integrated with internally displaced persons and the host

population. Around 13,000 also live in the camp of Sayam Forage, which is the main site

included in this project.

- In the south-west of the country around 56,000 refugees from Mali live in three camps

(Tabareybarey, Mangaize and Abala) as well as Intekane, a “Zone d’Accueil de Réfugiés”

(ZAR), an area where refugees are able continue their semi-nomadic life. The project includes

the health centres in each of the four sites. The health partner for both regions in Niger is the

Association Pour le Bien-Être (APBE).

8

4.2. Key Evaluation Questions (KEQs) and Sub-questions (SUQs)

The evaluation will address the following headline questions. The analysis needed to answer them is

likely to touch on other possible sub-questions that may be further refined during the evaluation

inception phase.

Key Evaluation Question on relevance of the project design:

• KEQ 1: To what extent are the activities of the 2018-2019 project (baseline assessment, capacity

building of human resources, strengthening of health facilities, community-based programmes,

organizational learning activities, RH survey, etc.) relevant and appropriate for the overall goal of

the project to improve maternal, neonatal and contraception/family planning care services in

Cameroon, Chad and Niger?

o SUQ 1.1. How could the project design and the choice of activities be strengthened to

improve its relevance to reaching the goal of the project? How does this vary or not

across the different country contexts?

Key Evaluation Question on effectiveness of project implementation:

• KEQ 2: To what extent was project implementation in each country delivered as intended in the

following areas: capacity building of care providers, health facility strengthening, comprehensive

maternal and newborn care packages, comprehensive family planning services, best practices and

guidance to improve care? Did implementation vary between countries?

o SUQ 2.1. To what extent did the organisational set-up of the project, the tools and systems

used in the delivery of the project contribute to timely implementation?

o SUQ 2.2. What were the major factors influencing the implementation of the project as

intended? Did factors vary by context? If so, how?

o SUQ 2.3. Which enabling and challenging factors, if any, should be taken into account in

future program implementation? Why?

• KEQ 3: What were the costs of implementing key components of the project and the project across

the three contexts?

o SUQ 3.1. What were the main drivers of cost? Did this vary by context?

Key Evaluation Questions on contribution to results:

• KEQ 4: To what extent did the project reduce maternal and neonatal mortality and morbidity and

what were the major factors that influenced changes in mortality and morbidity in targeted

populations in Cameroon, Chad and Niger? To what extent did the project increase contraceptive

uptake? A detailed table of key outcomes is available in Annex 1.

o SUQ 4.1. What were the specific changes in health care workers’ maternal and neonatal

knowledge, beliefs, and practices and do they correlate with any changes in their maternal

and neonatal health? Why/why not? Did this vary between contexts? If so, how?

o SUQ 4.2. What were the specific changes in health care workers’ knowledge, beliefs, and

practices related to the counselling for and provision of contraceptives and did these

contribute to improved contraceptive uptake? Why/why not? Did this vary between

contexts? If so, how?

CSH/UNHCR/20 0106 - ToR Evaluation - UNHCR Saving maternal newborn lives (Final).docx 9/15

o SUQ 4.3. To what extent did the community-based programme contribute to the project

objectives? How, if at all? Did this vary between contexts? Why/why not?

o SUQ 4.4. What were specific changes in CHW (community health worker) knowledge,

beliefs and practices around maternal and newborn care and care-seeking and family

planning practices? Why/why not? Did practices vary between contexts? If so, how?

• Key Evaluation Question on sustainability:

• KEQ 5: To what extent does the project demonstrate sustainability in terms of the continuation of

key activities of value and sustaining results in each country?

o SUQ 5.1. What are major factors that affect the sustainability of activities and results

achieved from the project in each country?

o SUQ 5.2. What support will potentially remain necessary to maintain activities and

achievement, and/or to complement any element identified as missing or insufficient in

each context?

o SUQ 5.3. To what extent has UNHCR institutionalised these approaches to maternal and

neonatal health and family planning into their public health work globally? What other

actions could UNHCR take to better incorporate learning from this project into their public

health programming globally?

4.2 Approach and methodology

The methodology – including details on the data collection and analytical approach(es) used to answers

the evaluation questions – will be designed by the evaluation team during the inception phase and

presented in an evaluation matrix.

The evaluation methodology is expected to:

- Refer to and make use of relevant internationally agreed evaluation criteria such as those

proposed by OECD-DAC and adapted by ALNAP for use in humanitarian evaluations4.

- Employ a mixed-method approach incorporating qualitative and quantitative data collection

and analysis tools including the analysis of monitoring data – as available, by measuring

the following outcomes listed and detailed in Annex 1:

➢ Reduced maternal and neonatal morbidity and mortality

➢ Improved maternal and neonatal care practices of health care workers, including

uptake of specific practices such as kangaroo mother care

➢ Improved counselling for and provision and uptake of contraceptives

➢ Strengthened community-level activities such as improved capacity for pregnancy and

postnatal home visits

The evaluation team is responsible to gather, analyse and triangulate data (e.g. across types, sources

and analysis modality) to demonstrate impartiality of the analysis, minimise bias, and ensure the

4 See for example: Cosgrave and Buchanan-Smith (2017) Guide de l'Evaluation de l'Action Humanitaire (London: ALNAP) and Beck, T. (2006) Evaluating Humanitarian Action using the OECD-DAC Criteria (London: ALNAP)

10

credibility of evaluation findings and conclusions. It will try to document best practices and lessons

learned to improve maternal, neonatal and contraceptive/family planning care.

Qualitative data sources should include, but are not limited to key informant interviews and focus group

discussions with UNHCR and partner staff, medical providers, community health workers/traditional

birth attendants and community level stakeholders are relevant and caregivers targeted for the project.

Quantitative data sources should include use of secondary data sources, which are listed below. Since

there has been an abundance of survey data collected from refugee households, any further collection

of household surveys is undesirable. Brief Knowledge, Attitude and Practice type assessments with

health care providers would be possible.

Data and information sources including the following existing sources:

- Project background documents and training reports;

- UNHCR’s Health information system (HIS): data can be used to provide monitoring data

on service coverage and health outcomes;

- Facility registers and other routine data sources can be used to assess patient needs and

provide indications of program quality;

- Trimester reports from all health facilities supported through the project;

- Facility checklists that were used by UNHCR/partners to monitor facility capacity for

service provision;

- Standardized Expanded Nutrition Survey, which was used to monitor early initiation of

breastfeeding and exclusive breastfeeding as well as the proportion of pregnant women

reporting receiving iron and folic acid supplementation from ANC facilities.

- Birth, maternal and newborn death reports, stillbirth data and maternal death audits from

HIS data and maternal mortality reports completed by NGO health partners.

- Reproductive, Maternal and Neonatal population level survey data

- Baseline assessment in 2018.

4.3 Evaluation Quality Assurance

The evaluation consultants are required to sign the UNHCR Code of Conduct, complete UNHCR’s

introductory protection training module, and respect UNHCR’s confidentiality requirements.

In line with established standards for evaluation in the UN system, and the UN Ethical Guidelines for

Evaluations, evaluation in UNHCR is founded on the inter-connected principles of independence,

impartiality, credibility and utility, which in practice i.a. call for: protecting sources and data;

systematically seeking informed consent; respecting dignity and diversity; minimising risk, harm and

burden upon those who are the subject of, or participating in the evaluation, while at the same time not

compromising the integrity of the exercise.

CSH/UNHCR/20 0106 - ToR Evaluation - UNHCR Saving maternal newborn lives (Final).docx 11/15

The evaluation is also expected to adhere with the ‘Evaluation Quality Assurance’ (EQA) guidance,

which clarifies the quality requirements expected for UNHCR evaluation processes and products.

The Evaluation Manager will share and provide an orientation to the EQA at the start of the evaluation.

Adherence to the EQA will be overseen by the Evaluation Manager with support from the UNHCR

Evaluation Service as needed.

5. Organisation, management and conduct of the evaluation

The UNHCR Public Health Section will serve as the Evaluation Manager. They will be responsible for:

(i) managing the day to day aspects of the evaluation process; (ii) acting as the main interlocutor with

the evaluation team; (iii) providing the evaluators with required data and facilitating communication with

relevant stakeholders; (iv) reviewing the interim deliverables and final reports to ensure quality – with

the support of the Evaluation Service at HQ.

The Evaluation Team should comprise a senior team leader and 1-3 team members. The team is

expected to produce written products of high standards, informed by evidence and triangulated data

and analysis, copy-edited, and free from spelling and grammatical errors.

The language of work of this evaluation will be in English and French and the deliverables will be in

English.

5.1 Expected deliverables and evaluation timeline

The evaluation should be completed from Mid-February 2020 to September 2020 and will be managed

following the timeline tabled below.

The key evaluation deliverables are:

• Inception report with evaluation matrix, data collection toolkit (including questionnaires, interview guides, focus group discussion guides) and details on the analytical framework developed for / used in the evaluation;

• Preliminary findings sensemaking workshop with UNHCR Public Health Section

• Three country-level reports including recommendations (40 pages excluding annexes)

• One synthesis evaluation report including recommendations (20-30 pages excluding annexes)

• Power Point Presentation for purpose of dissemination of the evaluation findings

• Executive summary5

• All raw data (quantitative and qualitative) collected should be anonymised and provided to UNHCR Evaluation Service.

5 The evaluation ToR, final report with annexes, and formal management response will be made public and posted on the evaluation section of the UNHCR website. All other evaluation products (e.g. Inception Report) will be kept internal.

12

Activity Deliverables and payment schedule Indicative

timeline

Minimum # of

estimated days

Phase 1: Inception phase including:

- Initial desk review and key

informant interviews.

- Circulation for comments and

finalisation

Final inception report – including

methodology, refined evaluation

questions (as needed) and evaluation

matrix; logistics support requirements

for data collection.

Payment 30%

Mid to late

March 2020

21

Phase 2: Data collection including:

- In-person/virtual interviews with

UNHCR staff and partners (HQ and

countries)

-Field visits to Chad, Cameroon and

Niger

All secondary and primary data

collected across 3 countries and HQ.

Payment 30%

Late March-

June 2020

50

Phase 3: Data analysis and

sensemaking/validation workshop

including:

- Data analysis of each country

- Data analysis across countries

- Facilitate sensemaking/validation

workshop of preliminary

evaluation findings, conclusions

and proposed recommendations

Facilitation of a

sensemaking/validation workshop of

preliminary findings, conclusions and

recommendations with UNHCR HQ

Payment 20%

July 2020 21

Phase 4: Report drafting and

finalisation

- Draft evaluation reports per

country

- Draft synthesis evaluation

report

- Powerpoint presentation of

findings and

recommendations

- Finalise reports

3 country-level evaluation reports for

Cameroon, Chad and Niger with

executive summaries and

recommendations

1 synthesis evaluation report across

all 3 countries and HQ-level with

executive summary and

recommendations

1 powerpoint presentation of findings

and recommendations

All raw data anonymised

Payment 20%

Aug-Sept.

2020

45

6. Evaluation team qualifications, selection and contracting

The evaluation will be undertaken by a team of a team of independent consultants – an evaluation

Team Leader and 1-3 Team Members – selected by means of a competitive selection process. It is

considered important that the same 2-person team will go through all 3 countries for the evaluation; the

participation of the team leader in all country visits is not negotiable.

The evaluation consultants’ selection process will be carried out by the UNHCR Public Health Section

in cooperation with the Evaluation Service. In line with the UNHCR Evaluation Policy, prior to hiring the

consultant(s)/Evaluation Team, any actual or potential conflict of interest will be assessed.

Contracting will be via individual consultancy contract. Only proposals submitted by a team of

consultants will be reviewed, and each member of the team will be contracted separately.

CSH/UNHCR/20 0106 - ToR Evaluation - UNHCR Saving maternal newborn lives (Final).docx 13/15

Functional requirements for the individual consultants – who should both be able and willing to travel

to the selected sites are as follows:

Evaluation Team Leader

• Advance university degree in public health or related health field.

• A public health or clinical background (nurse, medical doctor) with a strong experience in maternal/newborn health care and contraception/family planning.

• At least 10 years’ experience in the area of maternal, newborn and child health and public health, preferably in programmes in low resource settings.

• Track record of experience in project evaluations, particularly multi-country evaluations.

• Strong expertise in both qualitative and quantitative data analysis and research methods.

• Proven experience in leading an evaluation team in challenging contexts.

• Experience in the formulation, monitoring and evaluation of MNCH projects

• Experience working with refugees and/or in humanitarian settings would be desirable.

• Familiar with costing RMNCH programmes and conducting cost-effectiveness analysis

• Excellent spoken, writing and reporting skills in French and English.

• Good communication skills

Desirable: understanding of the forward vision regarding the refugee context and assistance to

refugees, namely the Global Compact on Refugees, and inclusion and integration of refugees into

national policies, strategies and systems.

Evaluation Team Member

• University degree in public health

• Experience in maternal, neonatal and child health programmes in resource limited settings

• At least 4 years of experience in evaluating MNCH programming

• Experience in quantitative and qualitative data analysis and research methods.

• Experience in costing analysis, preferably of MNCH programmes.

• Excellent spoken and written French and English.

• Good communication skills

Desirable: additional experience in health economics.

6.1. Requirements of the proposal:

The proposal should contain the following:

• A technical proposal outlining a brief overview of the envisaged approach to the evaluation based on the ToR. This should reflect the team’s understanding of the purpose of the evaluation and key questions, specify the roles and responsibilities of the team members, the anticipated timeline and any preliminary analytical framework that would be used.

• CVs of team members

• Relevant sample report of an evaluation conducted by the proposed Team Lead

All proposals are due by February 3, 2020 11:59pm Geneva time. Proposals should be emailed

to hqphn@unhcr.org.

14

Annex 1: Outcome Matrix

ID Outcome/ Output

Indicator(s) Data Sources (If Applicable)

1

Quality, comprehensive essential newborn health care provided

NMR per camp/site (number of neonatal deaths per 1000 live births)

UNHCR Health Information System (HIS) and program reports (Niger)

1.1

Capacity of health providers and community based health workers on essential new-born interventions strengthened

Number and % of health providers and CHWs trained in essential newborn care

Training report; programme reports.

% of health facilities

providing KMC Facility checklists

% of babies < 2000 g

placed in KMC Facility checklists

1.2

Adequate new-born supplies, equipment and guidelines/protocols to health clinics have been provided

% of health facilities with functioning newborn bag and mask available

Facility checklist

% of HFs with KMC

wraps available Facility checklists

2

Quality comprehensive maternal health services improved

Maternal mortality (crude numbers)

UNHCR Health Information System (HIS) and program reports (Niger)

-

Stillbirth rate per camp/site (number of stillbirths per 1000 live and stillbirths)

UNHCR Health Information System (HIS) and program reports (Niger)

- - -

Average number of 7 signal BEmONC functions performed in the past 3 months

Facility checklist

2.1 -

Capacity building of human resources: training of the health providers on routine and emergency obstetric care

Number and % of qualified health providers trained on routine and emergency obstetrics

Training reports; programme reports

- 2.2 -

Adequate maternal supplies, equipment and guidelines/protocols provided to health clinics have been provided

% of essential equipment for childbirth care available in each health facility

Facility checklist

- - -

% of essential medications for childbirth care available in each health facility

Facility checklist

CSH/UNHCR/20 0106 - ToR Evaluation - UNHCR Saving maternal newborn lives (Final).docx 15/15

- - -

% of HFs have clinical protocols and guidelines available (*detailed reproductive health and HIV guidelines BSC-RH)

Balanced scorecard; facility checklist

3 - -

Quality comprehensive family planning services improved

Contraceptive Prevalence Rate

Survey Facility checklist Number of new users of modern contraceptive methods

- 3.1 -

Capacity building of human resources: training of the health providers and community based health workers on contraceptive counselling, contraceptive modern methods and sensitization and promotion strategies strengthen

• Number and % of qualified health providers (doctors and midwives) trained on contraceptives methods and counselling

Training reports; programme reports

- - -

Proportion of postnatal clients who are offered counselling on family planning

Survey

- 3.2 -

Adequate contraceptive maternal supplies, equipment and guidelines/protocols provided to health clinics have been provided

• % of modern contraceptives methods provided by health facility to clients over the last 3 months

Facility Checklist

- - -

% of health facilities fully equipped with modern contraceptive methods

Facility checklist

- - -

% of modern contraceptives and associated supplies available at each health facility

Facility checklist

- - -

• % of HFs have clinical protocols and guidelines in family planning available

Facility Checklist

4 - -

Understanding of the specific situation of the new-born, maternal health care and family planning services and barriers has been achieved

Baseline assessment reports, work plans, monitoring tools and final evaluation report

- - - Reproductive health

survey completed in selected sites