Evaluation of the “Improvement of Mother and Child health Services in Uzbekistan, phase II” programme Final Report Akaki Zoidze Annemarie Hoogendoorn Giorgi Pkhakadze 31 August 2015 “This evaluation is supported and guided by the European Commission and presented by the AEDES consortium. The report does not necessarily reflect the views and opinions of the European Commission.” EuropeAid/132633/C/SER/Multi Framework Contract Beneficiaries 2013 LOT 8 Health Specific Contract n° 2014/353750v1
Transcript
Evaluation of the “Improvement of Mother and
Child health Services in Uzbekistan, phase II”
programme
Final Report
Akaki Zoidze
Annemarie Hoogendoorn
Giorgi Pkhakadze
31 August 2015
“This evaluation is supported and guided by the European Commission and presented by the
AEDES consortium. The report does not necessarily reflect the views and opinions of the
Annex 4 -- List of documents and literature reviewed
II IMCHS II Final Evaluation Report –August 31 / 2015
Annex 5 – List of persons/organizations consulted
Annex 6 – CVs of the Evaluation Team
Annex 7 – Project Key Stakeholders
Annex 8 – Status of adoption of IMCHS I phase recommendations by IMCHS II
Annex 9 - Status of the Project assumptions and risks
Annex 10 – Status of the achievement of the Project results and performance target
III IMCHS II Final Evaluation Report –August 31 / 2015
Acronyms
AEDES Agence Européenne pour le développement et la santé
BABIES Birth Weight and Age at death Boxes for an Intervention and Evaluation System
BCC Behaviour Change Communication
CSP Country Strategy Paper
EC European Commission
ENC Effective Neonatal Care
EPC Effective Perinatal Care
ES Evaluation Study
ET Evaluation Team
EU European Union
EUDU European Delegation to the Republic of Uzbekistan
FGD Focus Group Discussion
GIZ Deutsche Gesellschaft für Internationale Zusammenarbeit – German Development Cooperation
GoU Government of Uzbekistan
IMCHS Improvement of Mother and Child Health Service
IDP International Development Partner
IMCI Integrated Management of Childhood Illnesses
IYCF Infant and Young Child Feeding
KfW Kreditanstalt für Wiederaufbau, German Development Bank
KII Key Informant Interviews
MCH Maternal Child Health
MNCH Maternal, Newborn and Child Health
NET Neonatal Equipment Training
NIP National Indicative Plan
NR Neonatal Resuscitation
PLA Participatory Learning and Action
PT The Project Team
ToR Terms of Reference
UNFPA United National Fund for Population
UNICEF United Nations Children’s Fund
WASH Water, Sanitation and Hygiene
WHO World Health Organization
IV IMCHS II Final Evaluation Report –August 31 / 2015
1 Executive Summary
Background
During the last decade Uzbekistan has made progress in a range of key economic and social indicators, including those related to the health sector. However, challenges remain in sustaining this progress and keeping on track to achieve the Millennium Development Goals (MDGs) by 2015. The Overall Objective of the European Union-financed “Improvement of Maternal and Child Health Services – II Phase programme (IMCHS II)” is to support Uzbekistan in meeting MDG 4 and MDG 5 with a focus on improving the quality of Mother and Child Health (MCH) care. The IMCHS II Specific Objectives are to support the Ministry of Health to increase the quality of mother and child health services and to increase the capacity of families to make informed choices about health and nutrition. The programme was launched in 2012 as a continuation of the IMCHS – I Phase (2008-2011) and is implemented by the UNICEF Country Office (CO) in Uzbekistan in partnership with the Ministry of Health (MoH) of Uzbekistan and the WHO country office. The expected results and planned activities of the IMCHS II programme are presented on Figure 1 on page 2.
Evaluation Objectives and Methodology
In May, 2015 the European Union Delegation to Uzbekistan (EUDU) has commissioned this independent final evaluation study of the IMCHS II with global objective “to “to provide decision-makers in the Government of Uzbekistan, the relevant external co-operation services of the European Commission and the wider public with sufficient information to make an informed judgement about the performance of the programme, paying particular attention to the expected impact of the programme’s actions against its objectives”.
The specific objectives include as defined in the Specific Terms of References are:
To provide feedback to the EUDU and its national counterparts on the soundness (defined as relevance, effectiveness, efficiency, sustainability), the expected impacts and the EU added value and coherence of the IMCHS II;
To extract general lessons learnt and recommendations aimed at further enhancing the public health reforms in Uzbekistan.
Considering these objectives, the ES results are expected to inform three target audiences: the European Commission (EC) and its institutions, the Government of Uzbekistan (GoU) and the wider public. The geographical scope of the evaluation study is 6 regions of the country where the ''Improvement of Mother and Child Health Services in Uzbekistan, Phase II'' (IMCHS II) programme is implemented. The evaluation period encompasses approximately three-year period from the programme launch in 2012 up to June 2015. The evaluation object includes two contracts executed under the IMCHS II programme: (1) the IMCHS II Project contracted to UNICEF country office in Uzbekistan (Project) and (2) supply of equipment to the Project contracted to Med Concept Europe (Supply Contract).
The evaluation was conducted by the AEDES Evaluation Team, which included three experts.
Overall methodological approach for the Evaluation Study was an exploratory and explanatory, holistic single case study. The detailed evaluation design was developed by the AEDES Evaluation Team (ET) during the inception phase in consultation with the Project management and the EU Delegation in Uzbekistan and further refined during the field visit conducted from July 8th though July 20th, 2015. The selected 25 evaluation questions were grouped across seven evaluation criteria (relevance, effectiveness, efficiency, sustainability prospects, the expected impacts and the EU added value and coherence of the IMCHS II). the ET used “change analysis” (comparing the Project indicators over the time, with the
V IMCHS II Final Evaluation Report –August 31 / 2015
baseline and targets specified in the Project’s logical framework) along with the elements of “Meta-analysis” (extrapolating the sound findings of other studies and evaluations).
The evaluation methods included: (a) documentary review of the IMCHS II related and sectoral reports, studies, surveys and other relevant documentation; (b) in-depth interviews with key informants; (c) individual and group interviews with various IMCHS II stakeholders, partners, health providers, and experts; (e) field/site visits to the selected health and training facilities in Tashkent and other two pilot regions (Republic of Karakalpakstan and Khorezm Oblast). More details on the evaluation methodology and limitations are provided in section 2 of the main report.
Key evaluation findings
The detailed findings are presented in section 4 of the main report. The evaluation findings show that the IMCHS II programme was highly relevant as it targeted the main causes of child mortality and attempted to surmount key barriers in population coverage with quality MCHN services with the evidence-based programmatic interventions, was well designed drawing on lessons learned and recommendations generated by the first phase of the Project and well aligned with national priorities and had high national ownership. The Project was satisfactorily effective in reaching its performance targets, particularly in strengthening the MoH and its three levels and in building health provider’s capacity in provision of medical services for the children and mothers in Uzbekistan. The IMCHS II programme had sound management and most likely has delivered good value for money. The IMCHS II programme, along with the other development partner implemented projects in the MCH sector has most likely contributed to the improved impact level outcomes for children reported in the period from 2011 to 2013. Considering the Project’s full alignment with national priorities and the strong national ownership, its sustainability prospect is assessed as very high. The Project also demonstrated high coherence with the national and the EU policy frameworks and the EU value added, along with good quality and effective visibility tools employed. Institutional strengthening and capacity development of the health providers were particularly strong components of the Project. The overall assessment is that nutrition within IMCHS II has been heavily tilted towards linking up with the capacity development work within the project.
The overall ratings assigned by the ET to each evaluation criteria are presented in Table 4 on page 31.
Key evaluation conclusions and recommendations
A key feature, which contributed to the success of IMCHS II, is that over the past years, the Government of Uzbekistan has been very committed to improve service delivery quality for Mother and Child Health. The addition of nutrition and more preventive approaches were as new components within IMCHS II is rated as highly relevant due to the nutrition’s direct links with child mortality and maternal health.
The health providers decreased resistance and improved acceptance of new knowledge and skills - ensured through the Project-supported continuous knowledge and experience exchange, study tours, high quality technical assistance in MNCH care - was a critical achievement of the IMCHS II that will pave way for further enhancement of the MNCH sector in the country.
The strong national ownership secured through effective coordination and continuous engagement of the Project’s key international stakeholders (EU, UNCIEF and WHO) with the national stakeholders at national and oblast levels has been key determinant for the achievement of the results and high sustainability prospects for main IMCHS II interventions.
The Programme did not deliver the significant share of MCH equipment under the Supply Contract affecting the overwhelmingly positive perception of beneficiaries on the benefits delivered by the IMCHS II Programme as a whole. The evaluation findings indicate that
VI IMCHS II Final Evaluation Report –August 31 / 2015
unsuccessful outcome of the medical equipment procurement process that instigated the disappointment among the national partners may have been avoided if more information on the national regulatory requirements were included in the tender dossier and if the tender would have been organized with more close involvement of the GoU and allowing more time for the retendering.
Following recommendations were derived from the evaluation:
For the European Commission
Consider close engagement of the host governments in the medical equipment procurement process in Uzbekistan, and possibly in other countries that have relatively complicated administrative/regulatory requirements for import and weak national equipment maintenance systems. Closer involvement of a host government may not only provide additional assurance for the potential bidders that they will be assisted in meeting any regulatory/administrative requirements, thus increasing the number of potential tenderers, which do not have country specific experience, but also may mitigate any potential dissatisfaction related to the transparency of a tender process.
Consider the inclusion of community-based promotion of healthy behaviours and adequate nutrition as one of the areas for strengthening the local civil society organizations and local authorities in the provision of social services under the objective 4 of Multiannual Indicative Programme 2014-2020 for Uzbekistan. This can be achieved by building on promising experience from the community-based pilots in using Mahallas and Women’s Committees for the promotion of the healthy behaviours and proper nutrition for vulnerable groups, such as children and women.
Consider planning for joint independent evaluations with UNICEF and/or other United Nation agencies for contribution specific contracts under the FAFA. This will reduce the administrative burden on EU country delegation, a project team and a beneficiary and deliver cost savings.
For the Government of Uzbekistan
Strengthen the efforts to ensure the sustainability of IMCHS II supported key interventions by:
Completing the institutionalization process for the Supportive Supervision developed by the IMCHS II by adopting a special ministerial decree standardizing the SS procedures and tools and defining the funding source for SS either through a special budget programme at the national or oblast levels or through the extra-budgetary funds available at the health facility level.
Supporting the cascade trainings in MNCH at rayon level through allocating dedicated funding from the national and oblast budgets and employing the Supportive Supervision mechanism to monitor the replication and quality implementation of these trainings.
Adopting the concept on medical quality improvement developed with the support of the IMCHS II and undertaking concrete steps for institutionalization and implementation of the certification/licensing and quality audit mechanisms that will include the establishment of necessary institutional structures and processes described in the concept.
Retaining the MCH Coordination Council as an effective mechanism for partner coordination in the MCH sector after the completion of the IMCHS II programme. The continued functioning of MCHCC may help to monitor the joint efforts in implementation of the recommended actions above and other critical MNCH interventions supported by the international and national partners.
VII IMCHS II Final Evaluation Report –August 31 / 2015
For UNICEF, WHO and other key international partners
Continue advocacy and provision of the policy advise and technical assistance to the MoH to support the institutionalization of the revised medical curricula for the under and post graduate education, Supportive Supervision and the medical quality improvement mechanisms.
Engage the GoU in the dialogue on the necessity of health financing reforms for MNCH services and the improved financial management of MCH health facilities. Addressing the financial access barriers to MNCH services will be essential to ensure the improved maternal and child health outcomes. WHO, UNICEF and the World Bank can join efforts in this endeavor and draw on their successful experience in health financing reforms in other countries in the region
Apply additional efforts to increase the sustainability prospects for community nutrition/PLA interventions. In order to create sustainability, it is necessary for the IMCHS II before the Project end to apply additional efforts to ensure buy in and identify a key owner for community nutrition / PLA interventions within key national stakeholders. In more longer-term perspective it would be justified for the international partners remaining active to seek promotion of MCH nutrition through organization of study tours, (international) conferences, etc.
1 IMCHS II Final Evaluation Report –August 31 / 2015
2 Introduction
2.1 Cooperation Context
Uzbekistan is a lower middle-income country in Central Asia with over 30 million population, out of which 2.8 million are children under 5 years of age. An estimated 63% of the population lives in rural areas. During the last decade Uzbekistan has made progress in a range of key economic and social indicators, including those related to the health sector. However, challenges remain in sustaining this progress and keeping on track to achieve the Millennium Development Goals (MDGs) by 2015. The GNI per capita (Atlas Method) has increased from 510 US$ in 2001 to 2,090 US$ in 20141. The share of population living below the national poverty line has decreased from 27.5% in 20012 to 16% in 20113. The life expectancy at birth according to the national statistics in 2014 is 73.1 years, however the World Bank estimates stand at 68.1 years4. There are also differences in officially and internationally reported maternal and child health indicators. The national figures for infant and under-5 mortality were respectively 10.1 and 14.1 per 1,000 live births in 2011, while UN interagency Group estimates for the same year were significantly higher at 44 and 52 per 1,000 live births. 2006 MICS in Uzbekistan (the most recent source of information) showed that under-nourishment is around 5%, primarily related to poverty. Stunting (19%) exists as well as a result of inappropriate young child feeding practices. There also is a major and growing problem of overweight and obesity (44%). For most nutrition indicators Uzbekistan is close to the average for the whole of Central Asia. Some of the key health system and community level barriers/bottlenecks remain that hamper further progress in achieving optimal health outcomes for children. Namely, financial and other demand barriers in accessing cost-effective maternal, new-born and child health services, particularly for the most vulnerable population groups; the suboptimal quality of care determined not just by the obsolete infrastructure, particularly in rural areas, and the often outdated diagnostic and treatment practices, but also by poor capacity of health managers to implement continuous quality improvement programmes, particularly at sub-national and district level; insufficient knowledge and skills of parents to ensure adequate child care and development at home and seek specialized care; and the overall poor capacity of health system to implement health promotion and disease-prevention programmes and ensure intersectoral collaboration to promote effective nutrition, WASH and other public health programmes. To address these bottlenecks Ministry of Health (MoH), UNICEF and European Union initiated the “Improvement of Mother and Child Health Service - Phase I” (IMCHS I) programme funded by the EU from 2008 to 2011. The project supported the implementation of the integrated package of newborn and child survival interventions along the continuum of Maternal, Newborn and Child Health (MNCH) care from pre-pregnancy to childhood through establishment of 17 training centres in 8 Oblast of the country and trained up to 17,000 health professionals in modern treatment practices for newborns and children. The second phase of the project represents nationwide scale up of cost-effective new-born and child survival packages in the remaining six regions of the country: Fergana, Bukhara, Khorezm, and Tashkent Oblasts, the Tashkent city and the Republic of Karakalpakstan and support to the MOH in health system strengthening for achieving sustainable results in the area of mother and child health in Uzbekistan.
1 The Word Bank Data 2014. http://data.worldbank.org/indicator/NY.GNP.PCAP.CD 2 Millennium Development Goals Report. Uzbekistan. 2015. Tashkent: Center for Economic Research 3 The Word Bank Data 2014. http://data.worldbank.org/indicator/NY.GNP.PCAP.CD?page=2 4 The Word Bank Data 2014. http://data.worldbank.org/indicator/SP.DYN.LE00.IN
2 IMCHS II Final Evaluation Report –August 31 / 2015
2.2 The IMCHS II Programme and its Logic of Intervention
The diagram describing the IMCHS II results chain, or the intervention logic is presented on Figure 1 below
Figure 1: IMCHS II Logical Intervention Diagram
Overall Objective of the IMCHS II is to support Uzbekistan in meeting the targets of the Millennium Development Goals numbers 4 and 5, with a focus on improving the quality of Mother and Child Health (MCH) care. The IMCHS II Specific Objectives are to support the Ministry of Health to increase the quality of mother and child health services and to increase the capacity of families to make informed choices about health and nutrition.
The logical framework shows good internal coherence and reflects adequately the evaluated programme. Sources and means of verification are clearly defined and
1.1. Support the MoH to establish and operate the
MCHSCC, its secretariat, and JMT.
1.2. Establish a formal certification system and
process for the MCH care institutions.
1.3. Elaborate and pilot quality improvement
mechanisms for the MCH care institutions.
1.4. Pilot the Health System Strengthening
activities in two pilot regions
1.5. Perform need assessment, elaborate
specification of essential medical equipment
for provision of quality EPC, NR&ENBC
services
2.1.1. Conduct training and supervision of healthcare
providers on maternal, newborn and child
survival packages as per the approved standard
protocols and guidelines.
2.1.2. Support the MoH to implement the newborn
survival, child survival, and nutrition modules in
graduate and postgraduate curricula for MCH
care professionals.
2.1.3. Train health care system managers to support
improvement of MCH care services
2.1.4. Establish a platform for discussion and exchange
of experience and dissemination of the best
practices on MCH care sector reform issues.
2.2.1 Establish community based behavior change
mechanism based on participatory learning action
approach (PLA) under the ownership of selected
Mahallas and primary healthcare institutions
(SVP).
2.2.2 Develop PLA modules, BCC materials and tools
which promote healthy behaviors, including
nutrition, national awareness, and visibility
activities.
2.2.3 Train patronage nurses and Mahalla Advisors
from Women’s Committees to act as facilitators
in implementation of community activities.
2.2.4 Create support groups for promoting healthy
behaviors and monitoring of outcomes of
community based BCC activities.
3 IMCHS II Final Evaluation Report –August 31 / 2015
external conditions and risk are adequately reflected. The IMCHS II targets three out of four key health system functions as defined by WHO, such as stewardship, resource generation and service delivery. The IMCHS II includes both a reform component of the Mother and Child health care system with activities supporting healthy behaviours among families is dealing with the institutional strengthening of the Ministry of Health, by providing the needed skills and operational capacity to the primary, secondary, and tertiary health care system in accordance with international (WHO) standards (Annex 1 Presents the original and revised IMCHS II Logical Frameworks).
The procurement and supply of the equipment for the perinatal centres was a subject of separate contract tendered by the European Union Delegation to Uzbekistan (EUDU) and contracted to Med Concept Europe. The specifications for the equipment to be purchased were elaborated through the technical assistance provided under the IMCHS II Project implemented through the contract with UNICEF country office in Uzbekistan (Activity 1.5 on Figure 1).
2.3 Objectives and Intended Audience of the Evaluation Study
Current Evaluation Study (ES) of the “Improvement of Mother and Child Health Service - Phase II” (IMCHS II) programme funded by the EU is commissioned by the EU Delegation to Uzbekistan (EUDU) with a global objective “to provide decision-makers in the Government of Uzbekistan, the relevant external co-operation services of the European Commission and the wider public with sufficient information to make an informed judgement about the performance of the programme, paying particular attention to the expected impact of the programme’s actions against its objectives”.
The specific objectives include as defined in the Specific Terms of References are:
To provide feedback to the EUDU and its national counterparts on the soundness (defined as relevance, effectiveness, efficiency, sustainability), the expected impacts and the EU added value and coherence of the IMCHS II;
To extract general lessons learnt and recommendations aimed at further enhancing the public health reforms in Uzbekistan.
Considering these objectives, the ES results are expected to inform three target audiences: the European Commission (EC) and its institutions, the Government of Uzbekistan (GoU) and the wider public.
2.4 The Scope and Object of the Evaluation Study
The geographical scope of the evaluation study is 6 regions of the country where the IMCHS II programme is implemented. The evaluation period encompasses approximately three-year period from the programme launch in 2012 up to June 2015. The evaluation object includes two contracts executed under the IMCHS II programme: (1) the IMCHS II Project contracted to UNICEF country office in Uzbekistan (Project) and (2) supply of equipment to the Project contracted to Med Concept Europe (Supply Contract).
2.5 Methodology of the Evaluation Study
Overall methodological approach for the Evaluation Study was an exploratory and explanatory, holistic single case study design. This approach allowed both retrospective and prospective analysis of the evaluation object, as accomplishing the ES objectives required performing both formative and summative evaluation. Using this approach, the evaluation team studied “how” and “why” and “to what extent” changes have occurred as a result of the Project interventions.
The detailed evaluation design was developed during the inception phase in consultation with the Project management and the EU Delegation in Uzbekistan and further refined during
4 IMCHS II Final Evaluation Report –August 31 / 2015
the field visit. The selected 25 evaluation questions, finalized during the inception phase, provided findings across all the seven evaluation criteria as defined in the ToR (relevance, effectiveness, efficiency, sustainability prospects, the expected impacts and the EU added value and coherence of the IMCHS II). The evaluation questions were selected from the questions presented in the ANNEX 2 of the ToR (see for ToR Annex 2) based (a) on the documentary review, including the analysis of the Project’s logical framework, availability of data from the routine statistics, the Project’s progress and monitoring reports, surveys and studies conducted during the IMCHS programme implementation, etc. (b) online and face-to-face meetings with the EU Delegation in Uzbekistan during which the ET established possible critical/priority questions that the Project partners (EU, UNCIEF, MoH) would like to be included in the evaluation. Selected evaluation questions compiled into the evaluation matrix included the explanatory comments on: a/ origin of the question and potential utility of the answer, b/ clarification of the terms used, c/ Indicative methodological design, d/ foreseeable difficulties and feasibility problems if any, f/ design tables that specify sub-questions (if relevant), the proposed indicators, information sources and analysis strategy for each question (See Annex 3).
2.6 Analysis Strategy and Tools
Considering the overall and specific objectives of the evaluation, the ET used “change analysis” (comparing the Project indicators over the time with the baseline and targets specified in the Project’s logical framework) along with the elements of “Meta-analysis” (extrapolating the sound findings of other studies and evaluations). Considering the constrained time frame for the evaluation, the ET mainly relied on (a) documentary review of IMCHS II related and sectoral reports, studies, surveys and other relevant documentation; (b) in-depth interviews with key informants; (c) individual and group interviews with various IMCHS II stakeholders, partners, health providers, and experts; (e) field/site visits to the selected health and training facilities in Tashkent and other two pilot regions (Republic of Karakalpakstan and Khorezm Oblast). The evaluation team answered the evaluation questions using information sources that are independent enough from the management of the EC support. The ET applied efforts to ensure balanced representation of the involved and not involved parties in the delivery of EC support. The involved insiders include the Project management, MoH senior officials, and the EU Delegation members. They served as key informants for in-depth interviews. The Project outsiders included end users (health providers), development partners, local authorities, and experts. Their views were captured through appropriate data collection techniques like individual and group interviews. In-depth interview and group interview guides were based on the evaluation matrix (see Table 1)
Table 1: Evaluation questions included in the in-depth and group interview guides
Evaluation Questions/sub-questions to be included
Key Informant Interviews Project Team All EQs EU Delegation UNICEF Senior Management
MoH (Senior Officials) All EQs except EQ14-15 MoH (regional and local), other national stakeholders
All EQs expect EQ4, EQ5, EQ6, EQ12-17, EQ20, EQ23-25,
International Development Partners All EQs except EQ7-18, EQ21 Group Interviews Health Providers EQ2-3, EQ5, EQ7, EQ8, EQ19, EQ20, EQ23, EQ24
5 IMCHS II Final Evaluation Report –August 31 / 2015
The information obtained from different sources was cross-checked and triangulated to arrive to the evaluation findings. Preliminary findings were presented to the Reference Group for validation on June 19, 2015. The received feedback was incorporated in the findings and conclusions presented in this report.
2.7 Limitations and Data Gaps
The key limitation encountered by the ET relates to the timing of this evaluation. The Project’s End-line Assessment, which is expected to provide important information for evaluating the Project’s success in achieving some of the performance targets and ultimately the impact has been launched by UNICEF only recently and the results were not be available for this evaluation. This has limited the ET’s ability to make judgments on some of the results achieved and the impact anticipated.
As anticipated and thus indicated in the inception report, performing the Focus Group Discussions with patients and health providers has proved impossible within the given timeframe and circumstances at the sites visited by the ET in the Project targeted Oblasts. Instead, the ET has conducted group interviews with health providers and has interviewed selected patients to obtain general understanding on possible changes in provider-patient relations that may have occurred as a result of the Project.
Obtaining comparable information for making judgment on the cost benefit and cost-effectiveness of the Project outputs per the methodology suggested also proved to be impossible, as no comparable capacity development projects implemented in the similar context have been identified for which the cost data was available.
2.8 Ethical and Quality Standards
The ET strictly adhered to the ethical standards of the evaluation specified in the EU Programme/Project Evaluation Guidelines5 and DAC Quality Standards for Development Evaluation6.
2.9 Evaluation Team
The evaluation was conducted by the AEDES Evaluation Team, which included three experts (see the distribution of responsibilities and short CVs in Annex 6):
3 Evaluation findings:
3.1 Problems and Needs (Relevance)
EQ 1. Were the Project objectives aligned with the national and health sector policy priorities of the Republic of Uzbekistan (RoU) throughout the implementation period?
The Project’s overall objective is harmonized with one of the key goals of the national development strategy: “The Welfare Improvement Strategy of the Republic of Uzbekistan for 2013-2015” (WIS) to “further enhance human development factors (education, healthcare, housing, public services and ecological preservation, etc.) alongside the further development and support of the social sector and a significant increase in the quality of public services.” The Project and its specific objectives and results are directly referenced in the respective sections of the WIS. 5 http://ec.europa.eu/europeaid/evaluation/methodology/guidelines/gbb_det_en.htm#01_01 6 http://www.oecd.org/development/evaluation/qualitystandards.pdf
6 IMCHS II Final Evaluation Report –August 31 / 2015
The Project’s objectives are also in concordance with the objectives outlined in two consecutive health sector policy documents defining the national MCH strategic priorities: (1) the President’s Resolution N1096 from 13 April 2009 “On Additional Measures for Protection of Maternal and Child Health and Formation of the Healthy Generation”, (2) a “National Nutrition Investment Strategy” developed in 2009 which included promotion of breast feeding, Vitamin A supplementation, flour fortification and salt iodization and (3) the President’s Resolution N2221 from 1 August 2014 “On the State Program for Further Improvement of the Reproductive Health of the population and the Protection of Maternal, Child and Adolescent Health in Uzbekistan for 2014-2018”. Moreover, the Project rendered significant assistance in elaborating the latter document. The documental review shows that the Project objectives defined during the design stage maintained consistency with the national and sectoral strategic priorities throughout the Project implementation period (The Welfare Improvement Strategy of Uzbekistan for 2008-2012, The GoU Action Plan on MDGs 2011-2015, etc.). The national significance of the Project’s overall objectives was further strengthened by the GoU declaring the years 2012, 2013, and 2014 as dedicated to maternal and child health issues. Currently a GoU Decree on Strategy to Improve the Nutritional Status is being developed (presumably will include a focus on overweight and obesity as risk factors for NCD).
The summary judgment of the ET on the judgment criterion and indicators for this evaluation question is presented below:
Judgment Criterion 1.1
Aligned with the with the objectives and strategic priorities of RoU at the time of the Project development and currently (qualitative judgment as “fully aligned”; “partially aligned” and “not
aligned”)
Fully aligned
Indicator 1.1a Degree of alignment of the Project’s overall and specific objectives with needs and priorities of RoU as stated in the national development strategy (ies) in the period from 2012 through 2015
High
Indicator 1.1b Degree of alignment of the Project’s overall and specific objectives with needs and priorities of RoU as stated in the Maternal and Child Health (MCH) national policy and strategy from 2012 through 2015
High
EQ2. Was the choice of the Project interventions and the implementation strategy based on a diagnosis of the needs and capacities of the country, its population and other Project targeted groups and addressed these needs throughout the implementation period?
The IMCHS II identification and action fiches contain comprehensive and thorough situation analysis of maternal child health issues in the country. They took stock of the achievements, lessons learned and the remaining challenges by the time of the IMCHS II inception. The only gap identified by the ET was that the needs assessment on nutrition was done late and in a step-wise approach (first the Knowledge Attitude and Practice survey, followed by the in-depth study) which meant that due to the time pressure the development of the PLA module already had to start prior to the availability of the full results of the KAP study on the knowledge gaps among the women and the Patronage Nurses.
During the inception phase, the Project’s logframe was reviewed and some activities and their timeframes were revised to ensure better compliance with Procedures and Practical Guide (PRAG) for the EU External Actions and to account for implementation delays that occurred in the inception period. Most significant change noted by the ET relates to the Project Activity 1.2, which was changed to “Establish a formal certification system and process for the MCH care institutions”. The Description of Action (DoA) proposed that “...around 280 model SVPs (20 from each oblasts), 25 children polyclinics, one paediatric Hospital or ward whichever is available (around 100) and all maternities, multi-profile children hospitals and perinatal centres are certified as maternal and child friendly institutes”. According to the IMCHS II inception report, the UNICEF CO in Uzbekistan was supposed to do the certification work. This was not in line with the MoH practices, and neither the MoH
7 IMCHS II Final Evaluation Report –August 31 / 2015
nor the MCH care institutions were likely to allocate the necessary resources for the implementation of such approach. Moreover, the workload related to certification of this number of MCH institutions was unrealistically high for the project implementation. The new formulation of Activity 1.2 recommended that MoH would take the ownership of the certification process, while the project will provide technical assistance in elaboration of necessary documents and piloting the certification activities at selected MCH care institutions. The review of the Project’s inception and progress reports and the Mid-term Review show that the Project interventions were also modified during the actual implementation as a result of the Project’s Baseline Assessment (2013) and internal Mid-Term Review (2014).
The Project supported evidence-based interventions, such as the Effective Perinatal Care (EPC), Neonatal Resuscitation and Effective Neonatal Care (NR/ENC), BABIES Hospital, ambulatory and community based Integrated Management of Childhood Illnesses (IMCI), Infant and Young Children Feeding (IYCF) and Growth and Development Monitoring (GDM), Near Miss Case Review (NMCR), Nutrition (including through the Participatory Approach and Learning - PLA) with proven effects on maternal and child health outcomes7. These interventions targeted the leading causes of maternal and child mortality and morbidity (see Table 2) and were aligned across the care continuum for maternal, newborn and child health (see Figure 2).
Table 2: IMCHS II interventions and leading causes of neonatal and child mortality in Uzbekistan
Causes of Mortality8 IMCHS II interventions targeting the causes Neonatal 0-27 days
Perinatal Conditions - 68% of all neonatal deaths
Well Targeted. Perinatal conditions (prematurity – 38% of all neonatal deaths and birth asphyxia and birth trauma – 30% of the neonatal deaths) as a number one cause of a neonatal mortality were targeted by IMCHS II through the set of interventions supporting the implementation of the Newborn Survival training package. The Project interventions aimed at developing policy and legislative framework, assessing the quality of perinatal services, supporting the regionalization of the perinatal services and promoting the delivery of the Effective Perinatal Care (EPC) and Neonatal Resuscitation and Effective Neonatal Care (NR/ENC) and provision of Neonatal Equipment (Supply Contract) and Training in the use of Neonatal Equipment (TNE). This covered essential midwifery, obstetric and neonatal care, as well as a number of areas of special care, such as pre-eclampsia, postpartum haemorrhage, perinatal asphyxia, infection control and timely provision of the relevant level hospital care for mothers and newborns. The Project also provided support to further strengthening of the application of the BABIES tool and International Life Birth Definition (ILBD). IMCHS II has not supported interventions on an antenatal level that are directed towards the prevention of the deaths from preterm birth and other pregnancy complications, which was under the UNFPA mandate. Yet, IMCHS II supported policy work on national maternal and child health policies/strategies, which among other issues included measures to improve antenatal care (E.g. President’s Resolution 2221). IMCHS II also planned to address perinatal causes of death through the training of PNs in 5 key health behaviours and PLA.
Congenital abnormalities - 12%
Not Targeted. Congenital abnormalities, the second leading cause of neonatal mortality, was not targeted by IMCHS II programme through specific interventions, as pre-conception health was not explicitly included as a focus area. However, inclusion of nutrition as a crosscutting issue in the Project, may be regarded as a programmatic intervention preventing deaths due to
congenital abnormalities9.
Neonatal Infections (sepsis, meningitis, pneumonia, tetanus and diarrhoea) – 15%
Well Targeted. Infections (including respiratory infections, pneumonia, diarrheal and parasitic diseases, meningitis and sepsis) are the third leading cause of neonatal mortality (0-27 days), and the first leading cause of post-neonatal mortality and child morbidity (aged 1-59 months). In the neonatal period, they can be avoided by treating maternal infections during pregnancy, ensuring a clean birth, care of the umbilical cord and immediate, exclusive breast-feeding. All of these measures were supported through the Child Survival package encompassing NR/NET, Hospital IMCI (H-IMCI), Primary Health Care IMCI (P-IMCI) and Community IMIC (C-IMCI), breastfeeding and other nutrition interventions, which were supported by IMCHS II.
Children aged 1-59 months Pneumonia – 21% of all deaths of children aged 1-59 months
Diarrhoea – 13%
7 Lancet Child Survival Series 2008 8 CHERG/WHO/UNICEF for distribution of causes of neonatal and under-five deaths (published in Liu et al, Lancet 2012) 9 WHO. Congenital Abnormalities. Fact Sheet N370. 2014. Available at http://www.who.int/mediacentre/factsheets/fs370/en/. Accessed on January 31, 2015.
8 IMCHS II Final Evaluation Report –August 31 / 2015
Meningitis – 2%
Injuries and accidents (10%), NCDs (10%) other Conditions (44%)
Not Targeted. Other conditions were not explicitly targeted by any of the IMCHS programme interventions
Figure 2: The Project interventions across the MNCH care continuum
Source: Modified from WHO’s “Care Continuum for RMNCH services”
The Integrated services across the care continuum can be provided by families and communities, through outpatient services, clinics and other health facilities. As presented on Figure 3, the Project activities were supporting the strengthening of MNCH across all three care delivery platforms (hospitals, PHC and community based services).
Figure 3: The Project interventions across MNCH care delivery platforms
As planned, the IMCHS II Programme supported activities intended to address the key barriers, or bottlenecks in scaling up these evidence-based interventions to address the remaining gaps in child morality and morbidity in Uzbekistan to achieve MDG4 and MDG5. Achievement of this was sought by (a) strengthening policy, legislative and institutional frameworks through the Project’s result’s area 1 – Institutional strengthening, activities 1.1 and 1.4; (b) capacity building of health workforce (results area 2 – capacity building, activities 2.1.1-2.1.4; 2.2.3; supplying the necessary equipment (activity 1.5 and Supply Contract), introducing the quality improvement mechanisms (activities 1.2 -1.3) and promoting healthy behaviours (activities 2.2.1-2.2.4). Although the Project did not apply any systemic efforts towards identifying and addressing potential financial access barriers to MNCH services, as a support to health financing reforms was not explicitly included in the Project scope.
No formal assessments of the absorption and implementation capacities of the implementing partners – UNICEF and MoH were done. According to the documental review and the key informants, this was not conducted for several reasons: (a) absorption of funds and implementation was managed by UNICEF, the UN agency with proven track record in implementing the large scale programmes globally; (b) the implementation experience of IMCHS I rendered a good understanding of the implementation capacity of MoH’s central, regional and local institutions involved in the Project implementation. As for other sub-recipients of the Project funds (for trainings, etc.), UNICEF regularly performs routine
9 IMCHS II Final Evaluation Report –August 31 / 2015
assessment of sub-recipients financial absorption and implementation capacity as part of its risk mitigation procedures.
The overall strategic approach adopted by the Project - along with the situation analysis and other surveys performed by UNICEF in the last several years - also draws from the evaluation of the IMCHS I. According to key informants, the evaluation of phase I provided important recommendations on capacity development approach, the need for the improved quality assurance and M&E mechanisms, and the need to include nutrition interventions in the future project that were taken into account while designing the IMCHS II. The documentary review confirms that most of the recommendations were taken into account while designing the IMCHS II (see Annex 5).
There is evidence that alternative options (including whether or not the community-level nutrition elements within IMCHS II should include a link to work done in the agricultural sector) were analysed/discussed between the EU Delegation, the Project Team and UNICEF during the design stage. It was finally agreed to primarily focus on preventive approaches and insert nutrition in the training packages for the “Participatory Learning and Action” (PLA) pilot. In the view of various interviewees met by the ET, the approach for the integration of nutrition within IMCHS II could have been stronger if it would have been based on an overall nutrition intervention model in Uzbekistan that also includes (multi-sectoral) nutrition policy development and links up with existing nutrition interventions. More specifically on the nutrition component as cross-cutting issue reflected in both results areas of the Project:
Although it would have been relevant under the first result area within the IMCHS II in the logframe there is not explicit reference to integration of nutrition in the advocacy work to MoH on normative documents for MCH care sector reform. No assessment was done on nutrition-related policy and strategy-related issues within Uzbekistan. From the perspective of promotion of nutrition as a key element within MCH this is a missed opportunity.
While the Project logframe for the first Result Area (Institutional Strengthening) does not have explicit reference to nutrition, it has been covered as part of the MCH training packages (‘new born and child survival modules’ in graduate and postgraduate curricula) which are the second sub-component under the first Result Area. No assessment was done of specific training needs on nutrition among health personnel at the various tiers. Training of Patronage Nurses was added in the Phase 2, whereas the Phase 1 was primarily focused on training of doctors and nurses on curative aspects.
The logframe contains an indicator on the institutionalization of the PLA approach on the Five Key Behaviours that includes promotion of a healthy balanced family diet and targets women with young children.
Rollout of the post-graduate MCH training package for PHC health professionals (General Practitioners and Patronage Nurses) based on the materials developed under Result Area 1. This comprises training modules on nutrition-related topics (growth monitoring, promotion of breastfeeding, appropriate infant and young child feeding) which are rated as very relevant given the need to upgrade the knowledge on these topics.
Development and piloting of the PLA module on Five Key Behaviours. During the evaluation it has been universally stressed that it important to educate mothers on good health and nutrition habits for young children. As noted above, in IMCHS II, the needs assessment on nutrition was done late and in a step-wise approach (first the KAP, followed by the in-depth study), which meant that due to the time pressure the development of the PLA module already had to start prior to the availability of the full results of the KAP study on the knowledge gaps among the women and the Patronage Nurses. It was noted by the ET that the logframe does not contain quantified targets on the number of PLA activities in the selected Mahallas. After the baseline survey, the OVIs were made complete with target indicators. Right choice to
10 IMCHS II Final Evaluation Report –August 31 / 2015
comprise both the amount of nutrition messages by General Practitioners (GPs) and Patronage Nurses (PNs) and nutrition behaviour (exclusive breastfeeding, minimum dietary diversity, etc.).
Judgment Criterion 2.1
Adequate needs assessment and/or situation analysis for establishing the needs, problems and capacities of MCH sector was performed (qualitative judgment: highly adequate; adequate; somewhat adequate; not adequate)
Highly Adequate
Indicator 2.1a Evidence of comprehensive needs assessment/situation analysis in the Project identification/formulation documents
Yes
Indicator 2.1b Number and type of baseline assessments conducted 2 (KAP, In-depth analysis for nutrition)
Indicator 2.1c Extent to which the leading causes of MCH mortality and morbidity were identified Fully Identified
Indicator 2.1d Extent to which the key barriers to the scale-up of the MHC services were identified Identified, except financial access
barriers
Indicator 2.1e Evidence of local absorption capacity assessment performed Not performed, as deemed unnecessary
Indicator 2.2.f Evidence of local implementation capacity assessment performed Not performed, as deemed unnecessary
Judgment Criterion 2.2
The Project’s interventions selected and the implementation strategy adopted address the needs and problems identified (qualitative judgment “fully address”; “partially address” and “not address”)
Partially Address
Indicator 2.2a Evidence of analysis of strategic options for defining the Project’s interventions and implementation strategy performed and its quality in light of the international experience
Yes, High Quality
Indicator 2.2b Evidence of justification of the recommended implementation strategy, and of management and coordination arrangements;
Yes
Indicator 2.2c Extent to which the Project interventions as planned target the leading causes of MCH mortality and morbidity
Target 83% of neonatal mortality
causes and 37% of child morality causes
Indicator 2.2d Extent to which the Project interventions as planned target the key barriers/bottlenecks in scale up of the MCH services
Target most of the relevant barriers
Judgment Criterion 2.3
The Project interventions selected and the implementation strategy take into account the lessons learnt and recommendations derived from the implementation experience of the IMCHS I Phase (qualitative judgment: yes, partially, no)
Yes, most of the recommendations are incorporated
Indicator 2.3 The share of recommendations and lessons learned from IMCHS I adopted/incorporated 85% of recommendations
are fully or partially adopted
Judgment Criterion 2.4
The Project interventions selected and the implementation strategy take into account the local absorption and implementation capacity (qualitative judgment: yes, partially, no)
Partially
Indicator 2.4a Evidence that local absorption capacity of the Project beneficiaries: MoH, health providers and community are taken into account during the design of the Project interventions.
Partially, based on the implementation
experience of IMCHS I, no formal assessment
Indicator 2.4b Evidence that local implementation capacity of the key Project partner - MoH are taken into account during the design of the Project interventions and implementation strategy.
Partially, based on the implementation
experience of IMCHS I, no formal assessment
EQ3. Were all relevant stakeholders identified and involved in a balanced manner in the Project design, implementation and monitoring, including beneficiaries?
11 IMCHS II Final Evaluation Report –August 31 / 2015
The review of the IMCHS II Identification Fiche, DoA and the Project’s Inception Report and the interviews with key informants evidence that all the relevant stakeholders were identified and were involved in the design, including the international (WHO, UNFPA, KfW, GIZ, and the World Bank) and the national stakeholders - Ministry of Health, the National Coordinating Unit, National and regional MCH health institutions and under and postgraduate medical education institutions, etc. (see Annex 7 for description of key Project stakeholders).
Most of these stakeholders were engaged in IMCHS I through appropriate partnership arrangements. The only new stakeholders identified were for the Project’s community based interventions and included Mahallas and Women’s Committees.
While identified correctly, as evidenced from the Project’s Inception Report, there were only limited consultations with community outreach organisations, such as Mahallas and Women’s Committees, at the design stage. Furthermore, according to the UNFPA representative, potential duplication with UNFPA could have been avoided if better consultation was done during design stage. Also, according to a key informant no consultation with UNFPA on the managerial trainings was conducted. However, according to UNICEF staff and the presented documental evidence, extensive consultations with UNFPA were conducted at the design and inception phases of the Project and effective coordination mechanisms were in place throughout the Project implementation that should have provided ample opportunities to UNFPA representatives to put forward and resolve any coordination issues that may have had occurred. The ET paid special attention to the evidence on consultation process for the nutrition issues that were newly added (not included in the IMCHS I) crosscutting component of the Project. Key informant interviews revealed that during the inception phase, the relevant stakeholders on nutrition in Uzbekistan were all extensively consulted. In particular WHO (nutrition officer) has been providing a lot of useful suggestions on how to insert the nutrition components within IMCHS2 II. Overall, in the consultations the main focus was on how to shape the community pilot whereas it would have been useful to also focus on development of a common understanding at national level on the what, why and how of nutrition interventions within MCH, and to identify one or more nutrition champion(s) in Uzbekistan. Further consultation of key stakeholders on nutrition was done during project implementation, among others with various Departments and Institutes under MoH and WHO for development of the training materials and with a wider range of organizations including from the agriculture sector as part of the in-depth study. However, the results of the in-depth study only became available halfway into the project when the PLA module and intervention approach already had been defined.
The Project Identification Fiche and DoA contain the evidence that analysis of gender and vulnerable groups (e.g. children in poverty) were conducted at the design stage of the IMCHS II.
Representatives of all stakeholders interviewed reported that appropriate mechanism was put in place to ensure their balanced participation through the Maternal Child Health Coordination Council (MCHCC). According to the Project Team (PT), the optimized distribution of training responsibilities among the international partners (UNCIEF, GIZ and the World Bank) resulted in saving that were used to expand the scope of the capacity development. Another positive example of coordination was KFW providing equipment to the paediatric facilities in the targeted Oblasts and the Project providing training for the medical personnel in the use of this equipment as requested by the MoH. Yet, one key international partner (UNFPA) stated that the level of coordination on the issues relating to the maternal health left a room for improvement.
Judgment Criterion 3.1
All relevant stakeholders were identified using the rigorous approach: stakeholder analysis, gender analysis and analysis of the vulnerable groups (qualitative judgment: fully, partially, no)
Fully
Indicator 3.1a Evidence of stakeholder, gender and vulnerable groups analysis conducted and its correspondence to the accepted analytical approaches
Yes
12 IMCHS II Final Evaluation Report –August 31 / 2015
Indicator 3.1b Number and share of all key stakeholders and target groups identified 8 Stakeholder Groups (100%)
Judgment Criterion 3.2
Identified key stakeholders were involved in the design of the Project (qualitative judgment: fully, partially, no)
Partially (national stakeholders at the
local level and UNFPA were less
involved)
Indicator 3.2a Evidence that the key stakeholders identified were consulted during the Project design phase (minutes of meetings, stakeholder conferences etc.)
Partially
Indicator 3.2b Number of share of the identified key stakeholders that were consulted 6 out of 8 stakeholder groups
were consulted
Judgment Criterion 3.3
Appropriate mechanisms were put in place to ensure the key stakeholder participation in the Project implementation and monitoring (qualitative judgment: yes, partially, no)
Not performed, as deemed unnecessary
Indicator 3.3a Evidence of mechanisms for stakeholder participation in the Project coordination/implementation
Yes
Indicator 3.3b Evidence of mechanisms for stakeholder participation in the Project monitoring Yes (MCHCC meeting minutes, IDI)
Judgment Criterion 3.4
There was an appropriate balance of responsibilities between the various stakeholders during the design, implementation and monitoring of the Project (yes, no);
Yes
Indicator 3.4 The Respondents and ET judgment on the appropriate balance of responsibilities at all stages
Yes
EQ4. Was the analysis of the Project risks and assumptions valid and whether they remain valid at the time of the evaluation?
The Project’s risks and assumptions remained valid and unchanged throughout the Project implementation. Most of the assumptions appear to have been realized, while risks have not materialized (see for details Annex 6)
Judgment Criterion 4.1
The Project risks and assumptions were valid at the time of the Project design (qualitative judgment: yes, partially, no)
Yes
Indicator 4.1a The Evaluation Team (ET) judgment on risks and assumptions Valid. All assumptions and risks were valid at
the time of the Project design
Indicator 4.1b Share of risks and assumptions realized Up to 77% of all assumptions were
realized, 0% of risks realized
Judgment Criterion 4.2
The Project risks and assumptions are valid at the time of the Project Evaluation (qualitative judgment: yes, partially, no)
Yes
Indicator 4.2a Expert judgment on risks and assumptions Valid
Indicator4.2b Share of risks and assumptions modified 0
EQ5. Have the recommended monitoring and evaluation arrangements been appropriate and performed as planned?
The M&E arrangements were designed in accordance with Financial and Administrative Framework Agreement between the European Commission and the United Nations (FAFA) and the EU and UNICEF guidelines, however, the EU’s Results-Oriented Monitoring (ROM) system was not applied as prescribed, - only one ROM visit was conducted in September,
13 IMCHS II Final Evaluation Report –August 31 / 2015
2013, instead of the planned three (one ROM annually). In addition, the 2013 ROM mission did not look into nutrition, which hardly had started at that time. The reasons why the remaining two ROM visits were not performed appear to be beyond the control of the EUDU. The Project monitoring system was integrated with the monitoring system used by UNICEF globally and functioned as planned. There is evidence that monitoring results were used to correct the Project implementation course in at least one occasion: the monitoring showed the need for training the medical personnel in the use of equipment provided by other donor to the MHC facilities in the targeted Oblasts. In consultation with MoH, the Project initiated support to this training. The Joint Monitoring Team has performed the monitoring in accordance with the annual work plans. The Supportive Supervision (SS) was in place for both managerial and clinical trainings. As recommended by the IMCHS I final evaluation, the SS manual was developed and followed to the extent possible for the supervision of MCH clinical services. The regional and local stakeholders interviewed noted the usefulness of the Joint Monitoring System and particularly that of the SS.
UNICEF as planned, although with certain delays, conducted the Baseline Assessment and Mid-term evaluation. There is a documental evidence that the results of the Baseline Assessment in general validated the Project objectives and also informed the Project implementation. The Baseline and in-depth study have been highly instrumental in defining baseline and target indicators for the Project’s results and activities and designing the PLA pilot. The M&E plan included in the Project’s Description of Action (DoA) stipulate that along with this final evaluation commissioned by EUDU, an independent final evaluation is planned to be conducted by UNICEF before the end of the Project, for which UNICEF has already initiated a bidding process. The need to conduct two final evaluations of the Project were most likely determined by the EU and UNICEF regulations, yet, the ET notes that it may have been possible to agree on more cost-effective joint final evaluation option during the Project design.
Quantitative results on several key Project activities, including the PLA pilot data will become available through the end-line study, basically too late for project implementation correction but useful for drawing conclusions and lessons learned whether the approach has worked or not.
Judgment Criterion 5.1
The Project was monitored in accordance with EU and UNICEF M&E guidelines (qualitative judgment: yes, partially, no)
Partially
Indicator 5.1a Number of monitoring reports produced against the planned number 1 out of 3 ROM reports
Indicator 5.1b Evidence of monitoring results used for the Project implementation correction Yes
Judgment Criterion 5.2
The Project was evaluated in accordance with EU and UNICEF M&E guidelines (qualitative judgment: yes, partially, no)
Yes
Indicator 5.2a Number of evaluation reports produced against the planned number Partially
Indicator 5.2b Evidence of evaluation results used for the Project implementation correction Yes
Judgment Criterion 5.3
The Joint Monitoring Team (JMT) has performed the monitoring in accordance with the annual work plans (yes, partially, no)
Yes
Indicator 5.3a Evidence of annual workplans for JMT Yes
Indicator 5.3b Number of the JMT visits conducted against the planned number 10 out of 10
Indicator 5.3c Evidence of the JM results used for the Project implementation correction Yes
EQ6. Where the objectives formulated in clear and consistent manner and appropriate indicators of achievement (OVI) selected?
At the time of the evaluation, the Project objectives are formulated in a clear and consistent manner with a sound results chain. The initial results presented in the DoA were slightly
14 IMCHS II Final Evaluation Report –August 31 / 2015
reformulated during the inception phase to achieve more clarity. The Objectively Verifiable Indicators (OVIs) are present and realistic targets for them were defined after the results of the Baseline Assessment became available in 2013. However, the ET notes the absence of outcome/impact level OVIs on nutrition, which was included as an important cross cutting issue for the Project supported capacity development activities. Although it is acknowledged that there currently is no system for nationwide surveys on nutrition status and other means of data collection would thus be needed to be developed, it would have been logical to embed the focus on nutrition within the IMCHS II in the overall objectives through addition of nutrition impact indicators (“% of children 6-59 months of age with global acute malnutrition” and “(“% of children 6-59 months of age who are stunted”). The IMCHS II Identification Fiche recommended the use of anthropometric impact indicators from USAID’s Food and Nutrition Technical Assistance (FANTA) project as impact indicator but this has not happened.
Judgment Criterion 6.1
The Project objectives are formulated in clear and consistent manner (qualitative judgment: yes, no)
Yes
Indicator 6.1 The ET judgment on objectives Yes
Judgment Criterion 6.2
The Project’s OVIs are appropriate (qualitative judgment: yes, partially, no) Yes
Indicator 6.2a The ET judgment on appropriateness of the Projects OVI’s Yes
Indicator 6.2b Share of OVIs modified during the Project implementation 0
EQ7. What is the level of national ownership of the Project objectives and the Project supported interventions?
The evidence of high ownership of the Project’s objectives stems from the documental review and is supported by the interviews conducted by the ET. The same applies to the Project interventions planned under both Project components, with possible exception of the community based Participatory Learning Approach (PLA). The latter appears to have higher regional and local ownership expressed by the Oblast and Rayon health administrations, which may be explained by the pilot and “bottoms-up” nature of this specific intervention that may require some more evidence and demonstrated success to secure “buy-in” of the national stakeholders.
Judgment Criterion 7.1
The Project objectives and supported interventions are owned by the key national stakeholders (qualitative judgment: high, medium, low)
High
Indicator 7.1a Respondents and the ET judgment on the level of ownership High
3.2 Achievement of Purpose (Effectiveness)
EQ8. Have the planned benefits been delivered and received, as perceived by all key stakeholders (including women and men and specific vulnerable groups)?
The beneficiary representatives interviewed (MoH, other national and regional stakeholders, health providers and community representatives) reported that Project is meeting their expectations regarding the benefits delivered to each of these groups. The national level stakeholders are satisfied with the quality of the overall support provided by the Project through the policy advise, technical assistance and institutional strengthening activities. Health providers interviewed are satisfied with new knowledge and skills acquired through the Project-supported trainings. Community representatives and patients interviewed by the ET are satisfied with health services provided and consider useful the information supplied by health providers.
15 IMCHS II Final Evaluation Report –August 31 / 2015
The key national stakeholders (the MoH, the NCU) identified the failure to procure and supply all the planned MCH equipment for the Oblast Perinatal Centres as the foremost shortcoming of the IMCHS II Programme.
Judgment Criterion 8.1
The Project delivered the planned benefits to the key stakeholders and beneficiaries (qualitative judgment: yes, partially, no)
Partially*
Indicator 8.1a Evidence of the Project benefits delivered to the MoH (all three tiers) Partially*
Indicator 8.1b Evidence of the Project benefits delivered to health providers (doctors and nurses) Partially*
Indicator 8.1c Evidence of the Project benefits delivered to patients/community (including women and men and specific vulnerable groups)
Yes
Judgment Criterion 8.2
The key Project stakeholders and beneficiaries have a perception that they received the planned benefits (qualitative judgment: yes, partially, no)
Partially*
Indicator 8.2a The Project benefits received as identified by the MoH (all three tiers) Partially*
Indicator 8.2b The Project benefits received as identified by health providers (doctors and nurses) Partially*
Indicator 8.2c The Project benefits received as identified by patients/community (including women and men and specific vulnerable groups)
Yes
* Judgment criteria and indicators were assessed as “partially” due to the unsuccessful outcome of the perinatal equipment procurement and supply
EQ9. Did actual results match the performance targets set out initially (as defined in the Project’s Logframe?
By the time of evaluation, most of the targets for health providers and managers to be trained are already achieved/or exceeded.
By the end of the project, institutional strengthening and training activities as envisaged in the IMCHS2 Inception Report are expected to have been fully implemented and some targets even exceeded (e.g. for the training of personnel).
Some of the notable results and gaps include:
Under the Result Area 1
The MCHCC, its secretariat, and JMT have been established and are fully functional. 7 MCHCC meetings and 10 JMT visits have been organised covering various significant areas for MNCH;
Support was provided to MoH to elaborate and approve MNCH strategic document “On the State Program for Further Improvement of the Reproductive Health of the population and the Protection of Maternal, Child and Adolescent Health in Uzbekistan for 2014-2018”;
The legal framework for MNCH services was reviewed and 5 new decrees and 3 more decrees are submitted for approval. The national guidelines for EPC, NR, ENC, and IMCI were updated;
Under graduate curricula for doctors and nurses were updated and institutionalized, while post-graduate curricula is in the process of institutionalization in all medical education institutions throughout the country. For the under-graduate trainings there are 9 modules included in the curricula reflecting the main topics within MCH. For the post-graduate education the modules that already existed (USAID JSI) have been used and updated (for NR/ENC and EPC). Nutrition-related topics (BFHI, EBF, maternal nutrition) are covered rather scantily.
The tool for certification health facilities and methodology has been developed. Piloting of Four-Step approach and organization of 2-day TQM training in RoK,
16 IMCHS II Final Evaluation Report –August 31 / 2015
Fergana, and Andijan regions were conducted. Quality Assurance was done in 3 tertiary level maternity centres Tashkent.
With the WHO assistance, policy and tools for supportive supervision were developed, and a special MoH decree for institutionalization of the SS is under preparation and Assessment children’s rights in hospitals in Namangan (11 districts) has been implemented and recommendations provided;
The National Concept on Quality Improvement is finalized but not yet adopted. The national team for the implementation is trained. However is unclear whether the developed certification tool will be adopted and implemented.
Database of trainers and trainees on newborn and child survival packages is updated, tested, and used for project purposes. Database is also available at the Centre of Medical Education, however not yet institutionalized;
The specifications for the supply of equipment in line with MoH decree regionalization of perinatal services As part of the separate Supply Contract, 13 X-ray mobile units were purchased, installed and are in use in the perinatal centres nation wide. However, other equipment such as ventilators, monitors, etc., was not purchased due to the absence qualifying bids.
Under the Result Area 2
Total 760 national trainers, 1024 health managers and 11,334 health providers in Newborn and Child Survival packages were trained at national and oblast levels by May 2015; The average knowledge uptake is up to 70% as measured by the post-training tests. However, practical application of skills varies across disciplines and oblasts (see overall results in Annex 10). All training courses provided by the Project
were certified by the MoH and recognized at national level for career development10,
which in its term served as an incentive to be trained. Trainees were selected by the oblast Health Office to attend the training course that fitted their professional qualification. Although high attrition rates of health personnel are observed, particularly for health managers (according to key informants up to 40% of health mangers trained by the IMCHS I has been replaced by June 2015) affect may affect the sustainability of the achieved results.
As a result of the IMCHS II, the knowledge of health professionals on nutrition will have expanded through a series of training rounds for staff from SVPs in the six target Oblasts. This component targets family doctors, nurses and Patronage Nurses and trains them on Growth Monitoring and Development, Counselling on Exclusive Breast Feeding, and on Nutrition (appropriate infant and young child feeding and healthy family diets in general). It was noted by the team that there is limited focus on appropriate nutrition for women during pregnancy, as antenatal care was outside of the scope of the project. Nutrition during the lactation period is covered under the EPC and BF trainings.. Monitoring of the impact of the training of GPs and PNs on nutrition-related topics (“proportion of children whose caretakers have been counselled on young child nutrition”) was captured in the logframe but end-line results are not yet available.
In addition, a module on C-IMCI has been developed which formed the basis for training of 1,442 SVP staff - Patronage Nurses throughout the country. The C-IMCI module was developed by the Republican Medical Centre on Paediatrics and includes along with medical approaches, various nutrition-related issues. A pool of 100 master trainers covered all 14 Oblasts in a one year ‘crash training campaign. A mechanism was established for Supportive Supervision on C-IMCI. It is to be noted that this activity extends beyond the boundaries of the project as it was 100% funded
10 HPs are assigned to a category/level that contribute to determine their salary. In order to maintain or upgrade their category/level, doctors have to attend a minimum of 288 hours of professional upgrading courses within 5 years. Nurses are requested to attend a minimum of 44 hours within 5 years.
17 IMCHS II Final Evaluation Report –August 31 / 2015
by MoH. It is the main linking pin between the two sub-components under Result Area 2 within IMCHS II and can be seen as unintended positive impact of IMCHS II.
The community-level pilot on PLA based on the Five Key Behaviours including family diet diversification was developed and implemented in 12 selected pilot Mahallas (half urban, half rural) in two Oblasts (Namangan and Karakalpakstan). For various reasons the establishment of this activity has taken a lot of time so that actual implementation in the communities has been running for one year only. During the ET field visit to Chimbay District it became clear that most of the women groups that were formed have been reached through one session only. End-line results on “% of mothers who exclusively breastfeed their child 0-6 month of age” and the “% of children 6-23 months of age with at least a minimum dietary diversity” are not yet available.
Most likely, the outcome indicator targets characterising the accomplishment of the Project’s specific objective will be achieved/exceeded for the outcome indicators 1 (# of MoH normative documents on MCH care sector reform adopted or updated according to international standards) and 3 (# of Medical Institutes which have integrated at least 70% of the newborn and child survival modules in the graduate and postgraduate curricula). Prospects for fully achieving the targets for outcome indicators 2 and 4 are at best unclear at the time of this evaluation.
The prospects are not known for achievement of output targets for community based PLA approach due to the absence of the End-Line Assessment results (see Annex 10).
Judgment Criterion 9.1
Performance targets were defined for the specific objective and key activities of the Project (yes, partially, no)
Yes
Indicator 9.1 Shares of the OVIs with defined targets 100%
Judgment Criterion 9.2
Performance targets were achieved or will be achieved (yes, most likely, less likely, no) Most targets but not all are likely to
be achieved
Indicator 9.2a Extent to which the defined performance targets were achieved at the time of the evaluation Most were achieved
Indicator 9.2b The ET judgment on the prospects of achievement of the planned targets by the end of the Project
14 achieved/exceeded, or most likely to be achieved; 4 less
likely; 1 not likely and 6 not known
EQ10. Where there any unintended positive and negative results and whether the negative results could have been foreseen and managed?
The personnel was trained in the use of the modern MCH hospital equipment and while the equipment was not provided, some key stakeholders suggested that the trainings delivered have created demand for such equipment and facilitated adoption of modern clinical approached towards MCH care among the medical personnel previously resisting the change. Also, along with unintended positive result generated by the C-IMCI trainings mentioned above, the implementation of community based PLA practices may have strengthened general cooperation between the frontline medical providers and Mahallas.
Judgment Criterion 10.1
The Project had unintended positive and negative results (yes – positive, yes- negative, none)
Yes – Positive
No – Negative
Indicator 10.1 Evidence of unintended positive and negative results 3 possible unintended positive results identified
Judgment Criterion 10.2
If negative results established, they could have been foreseen and/or managed (yes, no) NA
18 IMCHS II Final Evaluation Report –August 31 / 2015
Indicator 10.2 The ET judgment on the possibility of foreseeing and/or managing the negative results NA
EQ11. Were any Project shortcomings due to a failure to take account of cross-cutting or overarching issues such as gender, environment and poverty during implementation ?
No Project shortcomings due to a failure to take account of crosscutting or overarching issues such as gender, environment and poverty during implementation were found by the ET.
Judgment Criterion 11.1
The Project had shortcomings in terms of under-performance on planned targets, unintended negative results and unforeseen risks (yes, no) No
Indicator 11.1
Evidence of the Project underperformance on targets, unintended negative results or unforeseen risks that have realized.
No
Judgment Criterion 11.2
If the Project shortcomings established, they are associated with failure in accounting for the cross-cutting or overarching issues (yes, no) No
Indicator 11.2
The ET judgment on the association of the Project shortcomings with cross-cutting or overarching issues.
No
EQ12. To what extent the intended results could have been achieved at a higher level of quantity / quality by changing for example:
a) Responsiveness and flexibility of project management
b) Monitoring of risks and external factors
c) Balance of responsibilities between the various stakeholders
d) Accompanying measures taken or to be taken by the government of the RoU?
In response to this question, the ET identified following:
Considering that no formal linkage was sought with the UNFPA programme on reproductive health that engages the PNs, - a better coordination with UNFPA may have resulted in better outcomes related to the antenatal care.
The translation of training materials for health managers to Uzbek language may have contributed to better knowledge uptake, as many young managers do not speak Russian;
It was acknowledged by UNICEF staff that the human resources approach that was chosen for the nutrition element within IMCHS II has led to sub-optimal results. While definitely it would have been more costly to employ a full-time international nutritionist at UNICEF, it is strongly believed that this would have been justified in order to create enough momentum and critical mass on nutrition. There has been a long debate on how to integrate nutrition in the project. EU wanted a full-time international nutritionist, UNICEF wanted to cover with their existing staff. Final agreement was that an international Nutritionist would be attached to the project on part-time basis. This has not been the most ideal way to insert a new component as there still is a need to work closely with GoU on sensitization etc. The project has benefited a lot from the presence of a WHO nutritionist although she was not formally part of the project structure. The idea for the PLA approach came from UNICEF, mainly on how to link the health sector with the community-level. UNICEF wanted to involve an NGO but MoH did not endorse that.
19 IMCHS II Final Evaluation Report –August 31 / 2015
Also, it would have been better to allot WHO a stronger formal footing on the nutrition work within the IMCHS II partnership, to more strongly engage the IMCHS II Health TA with the nutrition work which was primarily managed by UNICEF, and to link the project up with EC Nutrition Advisory Services in Brussels.
While the EUDU followed the EU public procurement rules and guidelines11 in preparing, announcing and executing the Supply Contract for the purchase of the medical equipment for perinatal centres, two key informants indicated that more involvement of the GoU and wider consultations with IDPs and local providers of the medical equipment in preparation of the medical equipment tendering process may have prevented the undesirable outcome of the tender (absence of qualifying bids for 8 out of 9 tendered items) organized for the supply of the medical equipment. Possible reasons of this undesirable outcome may have been (a) possible information gaps in tender dossier for potential tender participants on the national regulatory requirements for the import of medical equipment and the availability of local capacity for equipment maintenance; (b) low engagement by the EUDU of the GoU in the bid preparation and the tender process (beyond approving the technical specifications for the equipment and inviting the GoU representatives in the tender commission), which would have provided additional assurances for the potential tender participants that they would avoid logistical problems related to the administrative barriers (licensing, customs, etc.) and (c) impossibility to retender due to the limited time left until the expiration of the allocated funds for the equipment purchase. This finding is indirectly supported by the fact that one of the international partners (KfW) has received up to 70 bids and successfully conducted the international tender for the similar equipment that was supplied to paediatric hospitals in the country. Furthermore, it appears that closer involvement of the GoU agencies (MoH, NCU) in the tender process may have mitigated their current dissatisfaction that is partly related to the absence of detailed information on why the tender did not deliver the desirable results.
Judgment Criterion 12.1
The Project could have performed better in achieving the intended results (possible, not possible) Possibly
Indicator 12.1
The ET judgment on the possibility of achieving higher level (quantity and or/quality) of the intended results
Possibly
Judgment Criterion 12.2
If the plausible possibility is established, the improved results would have been achieved by (a) better responsiveness and flexibility of the Project management (yes/no); (b) improved monitoring of risks and external
factors (yes/no); (c) shifting balance of responsibilities between the
various stakeholders (yes/no); (d) Accompanying measures taken or to be
taken by the government of the RoU (yes/no).
a) Yes
b) No
c) No
d) Yes
Indicator 12.2
The ET judgment on the means for achievement of better intended results a) Yes - better coordination with UNFPA on antenatal care; human resources for nutrition
b) No
c) Yes – closer involvement of WHO in nutrition component
20 IMCHS II Final Evaluation Report –August 31 / 2015
in the Supply Contract
3.3 Sound Management and Value for Money (Efficiency)
EQ13. What was the quality of day-to-day management in:
a. operational work planning and implementation (input delivery, activity management and delivery of outputs),and management of the budget (including cost control and whether an inadequate budget was a factor);
b. management of personnel, information, property, etc,
c. whether management of risk has been adequate, i.e. whether flexibility has been demonstrated in response to changes in circumstances;
d. relations/coordination with local authorities, institutions, beneficiaries, other donors;
e. the quality of information management and reporting, and the extent to which key stakeholders have been kept adequately informed of project activities (including beneficiaries/target groups);
f. respect for deadlines?
Documental review shows that operational planning, budget management and cost control were conducted in accordance with FAFA and the contribution specific agreement and were highly adequate resulting in the savings that were used to expand the scope of the capacity development activities. The only problem mentioned by the Project Team was related to the fluctuation in the Euro exchange rate that has affected financial planning. However, considering FAFA, both sides (the EUDU and UNICEF) has limited flexibility in this matter. Management of the personnel and information appear to be adequate. According to the review of the Project documents and visibility tools, the Project management appears to have demonstrated the high quality of information management and reporting, and the extent to which key stakeholders have been kept adequately informed of project activities (including beneficiaries/target groups) were adequate by all accounts of the key informants interviewed. On top of the biannual progress reports specified in the contract, UNICEF provided EUDU and MoH with monthly updates on the Project implementation, which was considered as very useful by the key stakeholders.
While some delays in implementing the activities occurred (most notably for the hiring of the Team Leader, implementing the Baseline Assessment, etc.) the important deadlines were met, quantitative outputs in terms of trained managerial and clinical personnel were achieved or will be achieved before the end of the Project. One notable exception identified by the ET may have been the delay in announcing the tender for the purchase of the medical equipment. According to the EUDU this occurred due to the lengthy process of negotiations between the EU and UNCEF headquarters, which may have contributed to the undesired outcome of the equipment tendering process, as EUDU was not able to rebid because of the short time left for the use of the funds allotted for the equipment purchase under the supply component of the IMCHS II.
Judgment Criterion 13.1
Operational work planning and implementation (input delivery, activity management and delivery of outputs) and management of the budget (including cost control and whether an inadequate budget was a factor) for the Project was adequate (qualitative judgment: highly adequate, adequate, somewhat adequate, not adequate)
Adequate
Indicator 13.1
The ET judgment on the Project’s operational work planning and implementation and management of the budget
Good planning, satisfactory implementation and sound
21 IMCHS II Final Evaluation Report –August 31 / 2015
management of budget
Judgment Criterion 13.2
Management of the Project’s personnel, information, property, etc was adequate (qualitative judgment: highly adequate, adequate, somewhat adequate, not adequate)
Highly Adequate
Indicator 13.2
The ET judgment on the management of the Project’s personnel and property, etc
Both EUDU and UNICEF have followed the established
procedures and rules
Judgment Criterion 13.3
Management of risk has been adequate as demonstrated by the Project management’s flexibility in response to changes in circumstance (qualitative judgment: highly adequate, adequate, somewhat adequate, not adequate)
Not Applicable
Indicator 13.3
The ET judgment on the management of the Project’s risks none of the identified Project risks have materialized
Judgment Criterion 13.4
The Project management relations/coordination with local authorities, institutions, beneficiaries, other donors was adequate (qualitative judgment: highly adequate, adequate, somewhat adequate, not adequate)
Highly Adequate
Indicator 13.4
The ET judgment on the Project management’s relations/coordination Adequate
Judgment Criterion 13.5
The quality of the Project’s information management and reporting, and the extent to which key stakeholders (including beneficiaries/target groups) have been kept informed of project activities was adequate (qualitative judgment: highly adequate, adequate, somewhat adequate, not adequate)
Highly Adequate
Indicator 13.5
The ET judgment on the information management and reporting UNICEF’s information management system was highly satisfactory, The reporting exceeded pre-
agreed arrangements
Judgment Criterion 13.6
The deadlines for the Project activities and deliverables were observed adequately (qualitative judgment: always, in most cases, in certain cases, not observed)
In Most Cases
Indicator 13.6
The ET judgment on the respect of deadlines Delays were identified, in one case, the implementation delay
may have led to undesirable outcomes
EQ14. Extent to which the costs of the Project justified by the benefits delivered whether or not expressed in monetary terms (value for money) in comparison with similar projects?
As expected and specified in the inception report, the ET was unable to obtain comparable cost data to respond to this question. The detailed evaluation of the efficiency of resource use was constrained by two main factors:
Input Based Budget used by UNICEF – while all the relevant documents were
presented by the UNICEF CO, absence of the programmatic budget organization
wide, limited evaluators’ ability to analyse the resource use against the Project’s
component/activity outputs and outcomes.
Limited time and resources - Although there was a possibility to work out main costs
of interventions for further analysis and comparison with the costs of similar activities
supported by other donors, the performance of this exercise was not possible due to:
a) time and resource intensity of the task and the evaluation time constraints,
However, in overall, considering the achieved results and numbers of health personnel
trained, the ET considers that the Project has delivered adequate value for money.
22 IMCHS II Final Evaluation Report –August 31 / 2015
Although, as noted above, no specific financial analysis has been undertaken as part of this evaluation, it is assumed that efficiency levels for the elements of work on nutrition are very different for the two sub-components under the Result Area 2:
The integration of nutrition within the development of training manuals for health professionals (both graduate and post-graduate) has been piggybacked on to other capacity development activities already undertaken within IMCHS II. This has presumably led to efficiency gains. With relatively limited inputs from the Project side a set of manuals has been produced on various nutrition-related topics for both graduate and post-graduate training of health professionals.
On the other hand, the PLA pilot has been a stand-alone activity based on mahalla level cooperation between the PNs and the Community Advisors from the Women’s Committee and Mahalla foundation. There were limited synergies with other activities under the IMCHS II which all fell under the MoH. The development of the PLA model and consequent rollout in the selected pilot mahallas required considerable resources including separate baseline and end-line studies and close monitoring. This is due to the pilot nature of the work that was undertaken. Efficiency gains are to be reaped in future when the PLA model is being scaled up.
The human resources approach that was chosen for the nutrition element within IMCHS2 has led to sub-optimal results. While definitely it would have been more costly to employ a full-time international nutritionist at UNICEF, it is strongly believed that this would have been justified in order to create enough momentum and critical mass on nutrition.
Judgment Criterion 14.1
The Project has delivered the key outputs (trained personnel) at lower or comparable costs to other similar projects in similar contexts, including the comparable outputs in IMCHS I (yes, no).
Not Available
Indicator 14.1a Average costs per trainee for the Projects CD activities vs. the average costs per trainee for the similar projects
Not available
Indicator 14.1b Average costs per trainee for the Projects CD activities vs. the average costs per trainee for the IMCHS I.
Not Available
EQ15. Have GoU, UNICEF and WHO contributions been provided as planned? According to the documents reviewed (a) the GoU has provided the office space for the Project on he MoH premises (Tashkent Institute of Postgraduate Medical Education); (b) UNICEF has provided and exceeded its financial contribution requirement, and (c) WHO has fulfilled timely the contractual obligations stipulated under the UNCEIF subcontract.
Judgment Criterion 15.1
UNICEF’s was contribution provided in the planned period and in the amounts specified in the Project Contract (yes/no) Yes
Indicator 15.1
The ET judgment on the partner contribution The Financial contribution exceeded the commitment and was provided on-time
Judgment Criterion 15.2
GoU contribution was provided in the planned period and in the form specified in the Project Contract (yes/no) Yes
Indicator 15.2
The ET judgment on the partner contribution Provided as planned
Judgment Criterion 15.3
WHO contribution was provided in the planned period and in the form specified in the Project Contract (yes/no) Yes
Indicator 15.3
The ET judgment on the partner contribution Provided as planned
23 IMCHS II Final Evaluation Report –August 31 / 2015
EQ16. Have EU Delegation inputs (procurement and contracting) either direct or via consultants/bureaux provided as planned?
The EUDU inputs were fully provided for the IMCHS II Project, however significant share of the funds (up to 1,5 Million Euros from 1.7 million) committed for purchasing the medical equipment planned under the Project, were not disbursed due to the absence of qualifying bids.
Judgment Criterion 16.1
The EUD inputs were provided as planned (yes/no).
No
Indicator 16.1
The ET judgment on whether the inputs were provided as planned Yes for IMCHS II Project
No for Supply Contract
3.4 Achievement of wider effects (Impact) EQ17. To what extent the Project’s overall objective has been achieved as intended and whether the Project:
a. have been facilitated/constrained by external factors b. have produced any unintended or unexpected impacts and if so how have
these affected the overall impact. c. have been facilitated/constrained by project/programme management, by co-
ordination arrangements, by the participation of relevant stakeholders d. have contributed to economic and social development e. have contributed to poverty reduction f. have made a difference in terms of crosscutting issues like gender equality,
environment, good governance, etc.?
The baseline values and the latest available data for IMCHS II impact indicators are presented in Based on the Soviet live birth definition, used in the reported national statistics (until 1 July 2014) under-5 mortality fell from 14.1 to 12 per 1,000 live births between 2011 and 2013, and from 52 in 2011 to 43 in 2013 per 1,000 live births using the methodology of the UN Inter-Agency Group for Child Mortality Estimation. The infant mortality in the same period was reduced from 10.4 per 1,000 live births in 2011 to 9.8 cases in 2013 (see Table 3).. If national statistics is considered, Uzbekistan has reached and exceeded the MDG 4 and 5 targets. However, UN Inter-agency group estimates present different picture. Based on the Soviet live birth definition, used in the reported national statistics (until 1 July 2014) under-5 mortality fell from 14.1 to 12 per 1,000 live births between 2011 and 2013, and from 52 in 2011 to 43 in 2013 per 1,000 live births using the methodology of the UN Inter-Agency Group for Child Mortality Estimation. The infant mortality in the same period was reduced from 10.4 per 1,000 live births in 2011 to 9.8 cases in 2013 (see Table 3).
Table 3: Impact Indicators for IMCHS II Programme
Intervention logic Objectively verifiable indicators of
achievement
Progress towards logical framework
indicators, as of July 2014
Contribute to human
development in Uzbekistan
with special emphasis on
reducing child mortality
rates (MDG 4) and
improving maternal health
(MDG 5).
1. Infant mortality rate
Baseline: Less than 10.4 per 1.000 live
births (MoH, 2011)
44 per 1.000 live births (UNICEF,
WHO, WB, UNDP, Levels and Trends
in Child Mortality Report, 2011)
1. Infant mortality rate 9,8 per 1.000
live births (National official data,
2013).
1. Infant mortality rate 37 per 1.000
live births (UN Inter-agency Group for
Child Mortality Estimates, 2013).
2. Neonatal mortality rate
Baseline: Less than 6.3 per 1.000 live
2. Early neonatal mortality rate 4,9
per 1.000 live births (National official
24 IMCHS II Final Evaluation Report –August 31 / 2015
births (MoH, 2011)
23 per 1.000 live births (UNICEF,
WHO, WB, UNDP, Levels and Trends
in Child Mortality Report, 2011)
data, 2011).
2. Late neonatal mortality rate 1,4 per
1.000 live births (National official
data, 2011).
2. Neonatal mortality rate 14 per
1.000 live births (UN Inter-agency
Group for Child Mortality Estimates,
2013).
3. Maternal mortality rate
Baseline: Less than 23.1 per 100.000
live births (MoH, 2011)
28 per 100.000 live births(WHO,
UNICEF, UNFPA, WB Trends in
Maternal Mortality 2010)
3. Maternal mortality rate 20.2 per
100.000 live births (National official
data, 2013).
3. Maternal mortality rate 36 per
100.000 live births (UN Inter-agency
Group for Child Mortality Estimates,
2013).
4. Under 5 mortality rate
Baseline: Less than 14.1 per 1.000 live
births (MoH, 2011)
52 per 1.000 live births (UNICEF,
WHO, WB, UNDP, Levels and Trends
in Child Mortality Report, 2011)
4. Under 5 mortality rate 12 per 1.000
live births (National official data,
2013).
4. Under 5 mortality rate 43 per
1.000 live births (UN Inter-agency
Group for Child Mortality Estimates,
2013).
Source: MoH and MDG Report for Uzbekistan 2015
Given the programmes and interventions implemented in the regions focused on improvement mother and child health, regional disparities in child mortality rates decreased by 57.3% between 2000 and 2013. The most significant reduction of infant mortality was registered in Kashkadarya, Surkhandarya, Khoresm, Samarkand, Navoi and Fergana, where infant mortality rates – halved by 2013. The under five mortality rate also reduced substantially in Jizzak, Samarkand, Kashkadarya, Khorezm, Surkhandarya and the Republic of Karakalpakstan12.
While these impact level results cannot be directly attributed to the IMCHS II Project, it is likely that the Project, along with other key international stakeholders (the Word Bank, KfW, GIZ) has made contribution to these achievements (particularly for MDG 4) by assisting the GoU in systematically targeting the leading causes of child mortality through the evidence based interventions and in addressing key barriers to the effective coverage of MNCH services for the population of Uzbekistan. This in turn is expected to contribute to the economic development and poverty reduction in the country.
Due to the absence of the End-line Assessment results and the most recent child and maternal mortality data the ET is not in the position to respond in more detail to the evaluation questions on the Project impact.
Judgment Criterion 17.1
Positive changes in the OVIs measuring the achievement of the Project’s overall objective were achieved (substantial positive change, moderate positive change, incremental positive change, no change, incremental negative change, moderate negative change, substantial negative change)
Indicator 17.1c Maternal mortality rate Incremental to no change
Judgment Criterion 17.2
This change in OVIs can be associated with the Project’s contribution (qualitative judgment: associated/not associated)
Likely Association
Indicator 17.2 The ET judgment on Project’s contribution to the observed impact Likely association
Judgment Criterion 17.3
The Project’s overall effect was affected by the: (a) the Project management (facilitated, no effect, constrained); (b) by the coordination arrangements and participation of the relevant stakeholders ((facilitated, no effect, constrained).
Likely facilitated
Indicator 17.3 The ET judgment on the effect of the internal factors Likely facilitated
Judgment Criterion 17.4
The Project have contributed to economic and/or social development and poverty reduction (likely contribution, no contribution)
Likely
Indicator 17.4a Key economic and social indicators available for the Project targeted regions Not Available
Indicator 17.4b The ET judgment on the Project’s contribution to the economic/social development Likely
Judgment Criterion 17.5
The Project has made difference in terms of cross-cutting issues like gender equality, environment, good governance, etc (yes/no)
Not Available
Indicator 17.5 The ET judgment on the Project’s effect on cross-cutting issues Not Available
3.5 Likely continuation of achieved results (Sustainability)
EQ18. Whether the key national stakeholders have maintained the agreement with the Project objectives?
Findings from documental review and all in-depth interviews demonstrate that all key national stakeholders have been consulted and have maintained the agreement with the Project objectives throughout the implementation period.
Judgment Criterion 18.1
The key national stakeholders (MoH, regional and local authorities, academia, health providers) have been consulted in defining the Project objectives (yes, partially, no)
Yes
Indicator 18.1
Evidence of consultations with key national stakeholders on the Project objectives
Yes
Judgment Criterion 18.2
The key national stakeholders are in agreement with the Project objectives at the time of the evaluation (yes, partially, no)
Yes
Indicator 18.2
The ET judgment on the agreement of the key national stakeholders High
EQ19. How far the relevant national, sectoral and budgetary policies and priorities are affecting the project positively or adversely and what is the perceived level of support from government, health providers, academia and the general public (patients/community)?
As noted in the response to the EQ1, the Project objectives throughout the implementation period were and are fully aligned with the national development and strategic policies and priorities. According to the national key informants, the governmental declaration of the years 2012-2104 as years dedicated to maternal and child health also ensured the budgetary priority of MCH, as a result more investments to maternal and child health services were allocated at national and local levels. If this trend will continue and the provisions of the Presidential Decree 2221 on priority financing of the MCH services will be adhered, it is likely
26 IMCHS II Final Evaluation Report –August 31 / 2015
to positively affect the Project sustainability prospects. The level of support expressed by the representatives of government, health providers, academia and community, interviewed by the ET, towards continuation of the Project supported interventions was very high. However, there still is a need for advocacy and agenda building on MCH nutrition in Uzbekistan, also among participants within the MCHCC. The context is that in Uzbekistan nutrition for many years primarily has been narrowed down to a focus on micronutrient deficiencies and breastfeeding (the BFHI). Recently, the GoU highest-level attention for nutrition has increased however and the MoH in consultation with the relevant UN agencies is currently developing an action-oriented strategy.
Judgment Criterion 19.1
The Project is positively/adversely affected by the relevant national, sectoral and budgetary policies (positively/no effect/adversely) Positive
Indicator 19.1
The ET judgment on the effect of the policies Positively affected
EQ20. To what extent the Project is incorporated in the national budgets and embedded in local institutional structures and whether the MoH and other key national stakeholders have been properly prepared for taking over, technically, financially and managerially and what is the likelihood that they will be capable of continuing the flow of benefits after the project ends?
The ET found that (a) key coordination mechanism between the national and international partners in MCH are embedded in the institutional structure of the MoH in the form of the MCHCC; (b) enabling policy and legislative framework is created to sustain the key interventions (cascade training, revised curricula for medical education, Supportive Supervision) after the Project end; (d) the senior MoH representatives demonstrate high commitment in integrating these key interventions in the national budget. Also, the representatives of the oblast health departments and perinatal centres stated their intention to sustain the cascade trainings and SS activities through the oblast level budgets and extra-budgetary funds available to the health institutions. However, this commitment appears to vary from oblast to oblast and may depend on individual ownership retained by the oblast health departments and health providers. More specifically, some health authorities interviewed by the ET in one of the Project targeted Oblasts expressed their concerns about financial and organisational constraints that may hinder the replication of the cascade training activities without IMCHS II support in future, as financial resources allocated locally may not be sufficient to pay off national (travel expenses and per diem) and local trainers, as well as to ensure provision of needed training material. Yet, institutionalization of the training packages in the under and post-graduate curricula are expected to eliminate the need for the ad-hoc wide scale trainings of health providers in future.
Judgment Criterion 20.1
The GoRU is likely to take over the key activities of the Project that require continuation, once the Project ends (most likely/likely/less likely) Likely
Indicator 20.1a
Evidence of reflection of the Project supported priorities in the national budget
Not yet, however, likely to be reflected
Indicator 20.1b
Level of funding available for take over of the Project supported activities, once the Project provided support ends.
Adequate level of funding may be available
Indicator 20.1c
The ET judgment on the likelihood of adequate funding to become
available, once the Project support ends
Funding is likely to become available both at national, oblast and facility level.
However the level of commitment may vary
Judgment Criterion
The MoH is likely to maintain and effectively operate the MCH coordination mechanism established with the Project support (Most likely,
Most Likely
27 IMCHS II Final Evaluation Report –August 31 / 2015
20.2 likely, less likely)
Indicator 20.2
Respondents’ and the ET judgment on the likelihood of long term maintenance of the MCH coordination mechanism.
Likely to be maintained as long as the IDP support to MCH
continue
EQ21. How adequate is the project budget for its purpose particularly phasing out prospects?
According to the Project Team, utilizing the savings achieved with optimal coordination with IDPs and efficient budget management, the Project is addressing the identified gaps in the capacity development of specific target groups (both clinical and managerial trainings) which is expected to result in creation of the critical mass of the Master Trainers’ cadre at the national and oblast levels capable of carrying out the continuous education process on a sustainable basis. For the last oblast level training sessions, the Project is already limiting provided financial support, with MoH and oblast administrations filling in funding gaps (for e.g. participant’s travel, per diems).
Judgment Criterion 21.1
The Project will likely be able to finalize all the planned activities, including any planned handover measures (most likely/likely/less likely) Most Likely
Indicator 21.1a
Evidence of the handover measures in the Project budget Yes
Indicator 21.1b
The ET judgment on the likelihood of the adequate Project funding High likelihood, considering the savings achieved
EQ22. Have the financial sustainability, more specifically, the financial access to the Project supported MNCH services for the Project’s beneficiaries improved?
The reliable quantitative data is absent, however qualitative information obtained through the in-depth interviews with health providers and community representatives indicate that access to and affordability of the Project supported MNCH services for the Project’s beneficiaries have somewhat improved due to the following reasons: (a) better availability of medical equipment, drugs and skilled personnel at lower level (and less costly) facilities that are more accessible for the patients and their families (less travel and sustainment costs); (b) more rational prescription practices employed by the IMCHS II trained medical personnel, that reduces the cost of outpatient treatment for children and their mothers.
Judgment Criterion 22.1
Financial access to MNCH services at primary and secondary level has improved (improved/not improved/worsened) Likely have improved
Indicator 22.1a
% of the population that can not afford the necessary health services Not available
Indicator 22.1b
% of private expenditures on health Recent data is not available
Indicator 22.1c
Respondents and the ET judgment on financial access to the MNCH services Has likely improved
EQ23. Whether the knowledge, services and technologies in MNCH introduced and/or provided by the Project fits in with existing needs, culture, traditions, skills or knowledge and how well these were absorbed by the beneficiaries (health providers and families/communities)?
28 IMCHS II Final Evaluation Report –August 31 / 2015
The health providers were universally trained in the use of the equipment provided by IMCHS II and other IDP projects (e.g. KfW); By unanimous account of all health providers interviewed, the resistance and peer pressure observed during the IMCHS I among health providers in embracing and practically applying the transferred knowledge and skills have reduced. The JMT results show that the level of practical application of knowledge by the trained health providers has improved, and the likelihood of achieving the planned targets for the indicators characterising the improved practical skills is high for most of the indicators (see Annex 10). However, not all results for these and other indicators measuring the improved beneficiary (community and patients) knowledge are yet available due to the absence of the End-Line Assessment results.
Judgment Criterion 23.1
The knowledge provided through the Project supported trainings is appropriate (appropriate/less appropriate/not appropriate) and adequately absorbed by health providers (well absorbed/partially absorbed/not absorbed)
Appropriate and partially absorbed
Indicator 23.1a Results of the JMT Improved
Indicator 23.1b Respondents and the ET judgment on the appropriateness of the provided knowledge through the Project supported trainings
Acceptance of new knowledge and skills have
increased
Judgment Criterion 23.2
The patients/beneficiaries are satisfied with the MCNH services provided by the trained providers patients. (very satisfied, satisfied, not satisfied, not at all satisfied)
Not Available
Indicator 23.2 Level of patient satisfaction with provided services Not Available
Judgment Criterion 23.3
The patients/communities have improved knowledge of MNCH issues, including the those covered under the nutrition component (yes/no)
Not Available
Indicator 23.3 KAP survey indicators for which comparable time-series data is available Not Available
3.6 Mutual reinforcement (Coherence)
EQ24. Was the Project consistent with, and supportive of, the policy and programme framework within which the project is placed, in particular the EC’s Country Strategy Paper and National Indicative Programme?
The Project was generally aligned with one of the key overarching policy objectives of the EU relations with Uzbekistan: improve the living standards of the population. This policy objective was reflected in EC’s Central Asia Regional Strategy for 2007-2013 and DCI Indicative Programme for 2011-2013 by setting the goals pursuing the improvement of the social services. The maternal and child health improvement was considered as a key element (along with inclusive education) in achieving this goal. The Project is fully aligned with other global and regional strategies and policy frameworks joined by Uzbekistan such as MDGs, Convention on Rights of the Child, new UN initiative “A Promise Renewed” and WHO’s “Health 2020”.
In accordance with the objectives of the DCI Regulation (EC) No 1905/2006 of the European Parliament and of the Council, the Project aimed at the eradication of poverty in the context of sustainable development, including pursuit of the Millennium Development Goals (MDGs). Moreover, it aimed at the human development (article 4) addressing the essential needs of the population with prime attention to primary health, in particular by: (i) increasing access to and provision of health services for lower income population groups and marginalised groups, including women and children, with a central focus on the related MDGs, namely reducing child mortality, improving maternal and child health and sexual and reproductive health and rights as set out in the Cairo Agenda of the International Conference on Population and Development (ICPD), addressing poverty diseases, in particular HIV/AIDS, tuberculosis and malaria; (ii) strengthening health systems in order to prevent human
29 IMCHS II Final Evaluation Report –August 31 / 2015
resource crises in the health sector; and (iii) enhancing capacities particularly in areas such as public health.
With reference to the Central Asia Indicative Program (2007-2010), and in the framework of the Bilateral Cooperation, the Project was aligned with Priority Area 2: "Poverty reduction and increase of living standards in the context of the Millennium Development Goals" within the purpose of improving provision of and access to social services in the rural areas.
The EC Action Plan on Nutrition (2014) urges the scaling up of nutrition-sensitive interventions through incorporation of nutrition within other sector approaches (such as health, education, agriculture, fisheries/aquaculture, water and sanitation, social protection), both in programme design and through insertion of nutrition indicators in monitoring frameworks.
This has been through integration in the health sector through IMCHS II.
The EU Decision on Diversification of Agriculture (with emphasis on piloting of horticulture) signed in December 2012 could provide a good framework for integration of community-level health and nutrition education (e.g. based on the PLA approach as developed under the IMCHS II project).
The EU Decision on Sustainable Management of Water Resources in rural areas in Uzbekistan signed December 2014 (5-year programme) has the potential for piggybacking of further community-level nutrition work. In particular, this applies to the component on collaboration with Water User Associations to be indirectly managed by UNDP, which can provide a delivery mechanism for continuation and scaling up of PLA on the Five Key Behaviours at community-level through the Water User Associations as entry point. Logically, such community-level work should link up with the SVPs, the recently expanded PN outreach network (trained in C-IMCI) in particular.
According to the Commitments and Indicators of the Paris Declaration, the Project was on one hand, strongly based on the country’s national development strategies and priorities (National Indicative Plan Priority) in the areas of human capital development (alignment), and, on the other, it was conceived to promote in the long term the transfer of ownership on health care services to the national institutions. In line with the additional Indicators on Aid Effectiveness adopted by the EU, the Project followed the principle of avoiding the establishment of new project implementation units.
Judgment Criterion 24.1
The Project objectives, targeted beneficiaries, timing is consistent with and supportive of the EC’s Regional Cooperative Programme with Central Asia, Country Strategy Paper and National Indicative Programme for the RoU (fully, partially, not consistent/supportive).
Fully
Indicator 24.1 Degree of alignment with Central Asia DCI Indicative Programme, CSP, NIP objectives and priorities
Fully aligned with DCI, CSP and NIP
3.7 EU Added Value
EQ25. The Project was complementary and coordinated with the intervention of the EU Member States in the RoU. More specifically the Germany’s KfW and GIZ. There is evidence of a concerted effort between the Project and these partners in avoiding the duplication and fostering the synergies by combining the efforts in quality improvement initiatives in the pilot regions with GIZ and complementing the KfW equipment provision programme with necessary trainings provided by the Project to the medical personnel in the use of these equipment.
Judgment Criterion 25.1
The Project is complementary and coordinated with the intervention of EU Member States in the RoU (yes/no)
Yes
30 IMCHS II Final Evaluation Report –August 31 / 2015
Indicator 25.1 Evidence of complementarity with the EU member country intervention Yes (KfW, GIZ, the World Bank)
Judgment Criterion 25.2
There is a concerted efforts by EU Member States and the EC to optimise synergies and avoid duplication
Yes
Indicator 25.2 Evidence of coordination efforts Yes
4 Visibility
Drawing from the documental review, key informant interviews and site visits, the ET concludes that the visibility of the project has been ensured through a range of institutional and communications mechanisms:
Press coverage:
The official launch of the project on November 22, 2012 received wide media coverage.
There has been good media coverage of the Joint Monitoring Team visits (including the heads of mission for UNICEF, the EU Delegation and WHO in Uzbekistan) to the MCH training centre in Tashkent in December 2013.
Various press releases have been issued and project visits were organized for journalists, which resulted in publication of 55 news articles during the first two years of operation.
Interviews with the UNICEF Representative, the EU Ambassador and MoH officials were broadcasted on the main national TV and radio channels.
Institutional presence and visibility:
The establishment of a Secretariat for the MCHCC has helped to give the project a concrete presence in Uzbekistan. The MCHCC gives the Project increased visibility within all levels in the health sector, including the policy, advocacy and communication. The Secretariat also serves as the Project Steering Committee (PSC).
The Project office was opened on May 10 2013, which coincided with the celebration of Europe Day in Uzbekistan. This was well chosen and resulted in very wide coverage in the mass media (TV, radio, newspapers and reports).
In June 2013, an IMCHS II website has been established (www.ona-bolasalomatligi.uz) which is aimed at key stakeholders and target groups (e.g. it allows downloading of the training modules and educational materials) but also has an information function for the general public. Information is shared in English, Russian and Uzbek.
The key informants from EUDU expressed some concerns that the EU role in the MCH sector and in IMCHS II specifically was not always adequately promoted by the key implementing partners (UNICEF and MoH).
Communication mechanisms vis-à-vis the health sector:
The system of bi-annual MCHCC meetings serves as a platform for sector-wide dialogue and coordination. Seven MCHCC meetings were held so far.
Embedding of the project in the national health sector was also aimed at through the establishment of two technical working groups on programmatic and medical education issues.
Promotion materials
In the first year of operation, a full set of visibility materials on IMCHS II was produced in line with EU requirements within the contract. The materials are branded in line with the Uzbek national style and indicate that the project is based on a partnership
31 IMCHS II Final Evaluation Report –August 31 / 2015
between the EU, UNICEF and GoU. The materials include banners, business cards, doorplates, photographic panels, stands, wall plates, stickers and other items. In the second project year, a brochure was produced to provide a quick overview of project activities.
In the second year of operation, specific promotional materials were produced for distribution among health professionals and project partners (calendars, baseball caps, T-shirts, cups and pens).
5 Overall assessment
5.1 Project’s overall assessment
The evaluation findings show that the IMCHS II programme was highly relevant as it targeted the main causes of child mortality and attempted to surmount key barriers in population coverage with quality MCHN services with the evidence-based programmatic interventions, was well designed drawing on lessons learned and recommendations generated by the first phase of the Project and well aligned with national priorities and had high national ownership. The Project was satisfactorily effective in reaching its performance targets, particularly in strengthening the MoH and its three levels and in building health provider’s capacity in provision of medical services for the children and mothers in Uzbekistan. The IMCHS II programme had sound management and most likely has delivered good value for money. The IMCHS II programme, along with other IDP implemented projects in the MCH sector has most likely contributed to the improved impact level outcomes for children reported in the period from 2011 to 2013. Considering the Project’s full alignment with national priorities and the strong national ownership, its sustainability prospect is assessed as very high. The Project also demonstrated a high coherence with the national and EU policy frameworks and EU value added, along with good quality and effective visibility tools employed.
The overall ratings assigned by the ET to each evaluation criteria are presented in Table 4. The ET decided to assign separate ratings to the nutrition related activities of the Project, as the overall assessment for this cross cutting component notably differs from other MCH related “health” components of the Project.
Table 4: Overall assessment/rating of the IMCHS II programme
EU added value Satisfactory (no other nutrition projects)
5.2 Institutional Strengthening and capacity development
Institutional strengthening and capacity development of the health providers were particularly strong components of the Project. The Project was successful in creating enabling policy and
32 IMCHS II Final Evaluation Report –August 31 / 2015
legal framework and in institutionalization of the effective coordination mechanism for the country’s MCH sector and implementation of the functioning referral system for perinatal services. The critical mass of the national trainers was created and over 11,000 heath providers were trained by the time of the evaluation (including the refresher trainings). Most importantly, the national guidelines for the management of the most relevant MCH conditions and the under-graduate and post-graduate curricula for medical education were updated and institutionalized, thus creating the most important conditions for the sustainability of the IMCHS II results. The Project was less successful in building necessary capacity and ensuring the institutionalization of the certification and quality assurance mechanisms, as it remains unclear whether and when the quality assurance concept elaborated with the support of the Project will be developed into the actionable strategy and who will be the key driving force behind the implantation of certification/ licensing and other quality assurance mechanism in the country.
5.3 Nutrition as a crosscutting component of the IMCHS II
The overall assessment is that nutrition within IMCHS II has been heavily tilted towards linking up with the capacity development work within the project:
It was certainly relevant for the nutrition component to put emphasis on capacity development for health professionals. The achievements of the project with respect to the integration of nutrition in graduate and post-graduate training for health staff (medical doctors, nurses, and patronage nurses) can hardly be overrated as they are very substantial.
Capacity development at community-level was aimed at through the PLA pilot on promotion of the “Five Key Behaviours”. It is an innovative approach within the Uzbek setting to reach out to communities with health and nutrition messages through formation of women’s discussion groups. It is a very valuable approach that is more preventive in nature and focuses on promotion of healthy lifestyles including healthy diets. The latter in particular focuses on how to cover the specific nutritional needs during pregnancy and for young children. By the end of the project, the PLA approach will not yet have become institutionalised as further work will still be needed to ensure that there are cross-linkages between the MCH care provided through health service delivery channels (the Patronage Nurses in particular) and activities that are taking place within the community.
Within the project limited attention has been paid to further strengthening of the national-level understanding of the what, why and how for nutrition interventions as part of a balanced package of MCH interventions, and to establishment of mechanisms to ensure that in coming years nutrition will remain high on the GoU agenda.
As nutrition was piggybacked onto the IMCHS I project, the activities that were undertaken within this project primarily have stayed within the realm of the health sector. It proved impractical to promote agriculture-nutrition linkages and other food-based approaches) through IMCHS II, as was originally envisaged. Such approaches are more suitable for projects that already rooted in the agricultural, education, social and other sectors. They possibly provide new service delivery mechanisms.
6 Conclusions and Recommendations
6.1 Conclusions
The answers to the EQ1 through EQ 3 demonstrate that IMCHS II objectives were and are fully aligned with the national and health sector priorities throughout the Project implementation period. Moreover, the Project has contributed to the refinement of the national MCH strategic priorities. The IMCHS II Programme interventions and
33 IMCHS II Final Evaluation Report –August 31 / 2015
implementation strategy chosen were appropriate and adequate to achieve the desired objectives and were based on the experience and lessons learned from the IMCHS Phase I (Annex 8) and additional needs assessment and situation analysis. The Project attempted to address all key barriers to MCH services; the only missed opportunity within the Project was an absence of the systematic dialogue on MCH financing issues to mitigate risks affecting the Project outcomes. All the relevant stakeholders were identified and most of them were involved in the design of the Project. All stakeholders reported that appropriate mechanism was put in place to ensure their balanced participation. The findings for the EQ7 also show that there is a strong national ownership of the Project’s objectives and most of the Project supported interventions at all levels. These findings along with the high relevance of the IMCHS II, determine good sustainability prospects of the Project-supported key interventions once the EU funding will end in 2016.
By responding to the EQ4-EQ5 and, the ET concludes that the quality of the Project design was satisfactory. Annex 9 shows that the Project’s risks and assumptions remained valid and unchanged throughout the Project implementation. The Project’s monitoring and evaluation arrangements were appropriate, yet were not fully followed. In addition, the ET suggests that independent final evaluation jointly commissioned by EU/UNICEF may have been more efficient and cost-effective than two independent evaluations.
Analysis of the progress in achieving the Project outputs and results (Annex 10) conducted to answer the EQ9-12 concludes that the IMCHS II Programme appears to be delivering all the benefits (deliverables) planned under the Project and the beneficiaries appear to be receiving and acknowledging these benefits. The Project already achieved or on track to achieving important results and performance targets planned. Yet selected output and outcome level performance targets are unlikely to be met by the end of the Project implementation, or their prospects are unclear. The Project may also be delivering some unintended positive results. No Project shortcomings due to the due to a failure to take account of cross-cutting or overarching issues such as gender, environment and poverty during implementation were found by the ET.
Findings under the EQ 13 -164 demonstrate that overall, the quality of the Project’s day-to-day management was adequate. However, the Project could have performed somewhat better in achieving the intended results by better balancing responsibilities between the key partners and by possible accompanying measure taken by the GoU.
The EQ15-16 concludes that GoU, UNICEF and WHO contributions to the Project were provided as planned, while the EU inputs were fully provided for the IMCHS II Project, however significant share of the funds (up to 1,5 Million Euros from 1.7 million) committed for purchasing the medical equipment planned under the separate Supply Contract under the IMCHS II programme, were not disbursed due to the absence of qualifying bids.
The EQ17 responses suggest that the IMCHS II along with other IDP financed project’s in the MCH sector has most likely contributed to the improved child and maternal mortality and morbidity observed in the period from 2011-2013, however due to the absence of the more recent data and the End-line Assessment results, the ET is not in the position to make more detailed and sound conclusions on the IMCHS likely impact.
The answers to the EQ18-21 lead to the conclusion that the Project’s sustainability prospects are high and the Project’s budget appears to be adequate to successfully phase – out the Project. The reliable quantitative data is absent for responding to the EQ 22, however there are some indications that access to and affordability of the Project supported MNCH services for the Project’s beneficiaries have improved. The EQ 23 concludes that the knowledge, services and technologies in MNCH introduced and/or provided by the Project fits in with existing needs, culture, and traditions. Arriving to the judgment on how well the knowledge and skills were absorbed and applied by the beneficiaries (health providers and families/communities) is constrained due to the absence of the End-Line Assessment results.
34 IMCHS II Final Evaluation Report –August 31 / 2015
The responses to the EQ24-25 lead to the conclusions that the Project was consistent with, and supportive of, the national and EU policy and programme framework within which the project was placed and that the Project was complementary and coordinated with the intervention of the EU member state (Germany) in Uzbekistan.
Key conclusions and lessons learned
A key feature, which contributed to the success of IMCHS II, is that over the past years, the Government of Uzbekistan has been very committed to improve service delivery quality for Mother and Child Health. The addition of nutrition and more preventive approaches as new components within the Phase 2 of the Project is rated as highly relevant due to the nutrition’s direct links with child mortality and maternal health.
The health providers decreased resistance and the improved acceptance of new knowledge and skills - ensured through the Project-supported continuous knowledge and experience exchange, study tours, high quality technical assistance in MNCH care - was a critical achievement of the IMCHS II that will pave way for further enhancement of the MNCH sector in the country.
The strong national ownership secured through effective coordination and continuous engagement of the Project’s key international stakeholders (EU, UNCIEF and WHO) with the national stakeholders at national and oblast levels has been a key determinant for the achievement of the results and high sustainability prospects for the main IMCHS II interventions.
The Programme did not deliver the significant share of MCH equipment under the Supply Contract affecting the overwhelmingly positive perception of beneficiaries on the benefits delivered by the IMCHS II Programme as a whole. The evaluation findings indicate that unsuccessful outcome of the medical equipment procurement process that instigated the disappointment among the national partners may have been avoided if more information on the national regulatory requirements were included in the tender dossier and if the tender would have been organized with more close involvement of the GoU and allowing more time for the retendering.
6.2 Recommendations
The recommendations arising from the evaluation are divided in two groups: (1) for the European Commission and for the (2) key national and international implementing partners (MoH, UNICEF, WHO).
6.2.1 For European Commission
Consider closer engagement of the host governments in the medical equipment procurement process in Uzbekistan, and possibly in other countries that have relatively complicated administrative/regulatory requirements for import and weak national equipment maintenance systems. Closer involvement of a host government may not only provide additional assurance for the potential bidders that they will be assisted in meeting any regulatory/administrative requirements, thus increasing the number of potential tenderers, which do not have country specific experience, but also may mitigate any potential dissatisfaction related to the transparency of a tender process.
Consider the inclusion of community-based promotion of healthy behaviours and adequate nutrition as one of the areas for strengthening the local civil society organizations and local authorities in the provision of social services under the objective 4 of Multiannual Indicative Programme 2014-2020 for Uzbekistan. This can be achieved by building on promising experience from the community-based pilots in using Mahallas and
35 IMCHS II Final Evaluation Report –August 31 / 2015
Women’s Committees for the promotion of the healthy behaviours and proper nutrition for vulnerable groups, such as children and women.
Consider planning for joint independent evaluations with UNICEF and/or other United Nation agencies for contribution specific contracts under the FAFA. This will reduce the administrative burden on EU country delegation, a project team and a beneficiary and deliver cost savings.
6.2.2 For key national and international implementing partners
For the Government of Uzbekistan
Strengthen the efforts to ensure the sustainability of IMCHS II supported key interventions by:
Completing the institutionalization process for the Supportive Supervision developed by the IMCHS II by adopting a special ministerial decree standardizing the SS procedures and tools and defining the funding source for SS either through a special budget programme at the national or oblast levels or through the extra-budgetary funds available at the health facility level.
Supporting the cascade trainings in MNCH at rayon level through allocating dedicated funding from the national and oblast budgets and employing the Supportive Supervision mechanism to monitor the replication and quality implementation of these trainings.
Adopting the concept on medical quality improvement developed with the support of the IMCHS II and undertaking concrete steps for institutionalization and implementation of the certification/licensing and quality audit mechanisms that will include the establishment of necessary institutional structures and processes described in the concept.
Retaining the MCH Coordination Council as an effective mechanism for partner coordination in the MCH sector after the completion of the IMCHS II programme. The continued functioning of MCHCC may help to monitor the joint efforts in implementation of the recommended actions above and other critical MNCH interventions supported by the international and national partners.
For UNICEF, WHO and other key international partners
Continue advocacy and provision of the policy advise and technical assistance to the MoH to support the institutionalization of the revised medical curricula for the under and post graduate education, Supportive Supervision and the medical quality improvement mechanisms.
Engage the GoU in the dialogue on the necessity of health financing reforms for MNCH services and the improved financial management of MCH health facilities. Addressing the financial access barriers to MNCH services will be essential to ensure the improved maternal and child health outcomes. WHO, UNICEF and the World Bank can join efforts in this endeavor and draw on their successful experience in health financing reforms in other countries in the region.
Apply additional efforts to increase the sustainability prospects for community nutrition/PLA interventions. In order to create sustainability, it is necessary for the IMCHS II before the Project end to apply additional efforts to ensure buy in and identify a key owner for community nutrition / PLA interventions within key national stakeholders. Based on the experience so far, it seems logical to think that such institutional owners may be the Republican Paediatric Institute, which has been key in the development of the C-IMCI manual and/or Institute of Health and Medial Statistics responsible for the promotion of
36 IMCHS II Final Evaluation Report –August 31 / 2015
healthy life styles. In more longer-term perspective it would be justified for the international partners remaining active to also seek promotion of MCH nutrition through organization of study tours, (international) conferences, etc.
Annexes
37 IMCHS II Final Evaluation Report –August 31 / 2015
Annex 1. The Project Logical Framework (Logframe)
38 IMCHS II Final Evaluation Report –August 31 / 2015
Annex 2. Terms of Reference for the Evaluation of the "Improvement of
Mother and Child Health Services in Uzbekistan, phase II" programme
39 IMCHS II Final Evaluation Report –August 31 / 2015
Annex 3
RELEVANCE and Quality of Design
EQ1 Were the Project objectives aligned with the national and health sector policy priorities
of the Republic of Uzbekistan (RoU) throughout the implementation period?
Origin/Utility This question along with the EQ2 will help to assess several evaluation “topics”
included in the original ToR (a) the extent to which stated objectives correctly address
the identified problems and social needs; (b) extent to which the nature of the
problems originally identified have changed; (c) degree of flexibility and
adaptability to facilitate rapid responses to changes in circumstances; and (d) how
far donor policy and national policy are corresponding, the potential effects of any
policy changes; how far the relevant national, sectoral and budgetary policies and
priorities are affecting the project positively or adversely. The findings and
conclusions will be used to assess the relevance and sustainability of the Project.
Judgment Criterion 1.1
Aligned with the with the objectives and strategic priorities of RoU at the time of the
Project development and currently (qualitative judgment as “fully aligned”; “partially
aligned” and “not aligned”)
Indicator 1.1a Degree of alignment of the Project’s overall and specific objectives with needs and priorities of RoU as stated in the national development strategy (ies) in the period from 2012 through 2015
Indicator 1.1b Degree of alignment of the Project’s overall and specific objectives with needs and priorities of RoU as stated in the Maternal and Child Health (MCH) national policy and strategy from 2012 through 2015
Feasibility High
Methods Sub-questions:
1. Have the national development/sectoral priorities originally identified changed?
2. The extent to which the Project objectives have been updated in order to adapt to changes in the national/sectoral priorities?
EQ2 Was the choice of the Project interventions and the implementation strategy based on a
diagnosis of the needs and capacities of the country, its population and other Project
targeted groups and addressed these needs throughout the implementation period?
Origin/Utility This question along with the EQ1 will help to assess three evaluation “topics”
included in the original ToR (a) the extent to which stated objectives correctly address
the identified problems and social need and; (b) the extent to which the nature of the
problems originally identified have changed; In addition this question also addresses
40 IMCHS II Final Evaluation Report –August 31 / 2015
following evaluation topics specified in the ToR: (c) Adequate assessment of local
absorption capacity; (d) Adequate assessment of local implementation capacity; (e)
the quality of the analysis of strategic options, of the justification of the recommended
implementation strategy, and of management and coordination arrangements. The
findings and conclusions will be used to assess mainly the relevance of the Project.
The findings will also contribute to the assessment of the Project sustainability
prospects.
Judgment Criterion 2.1
Adequate needs assessment and/or situation analysis for establishing the needs, problems and capacities of MCH sector was performed (qualitative judgment: highly adequate; adequate; somewhat adequate; not adequate)
Indicator 2.1a Evidence of comprehensive needs assessment/situation analysis in the Project identification/formulation documents
Indicator 2.1b Number and type of baseline assessments conducted
Indicator 2.1c Extent to which the leading causes of MCH mortality and morbidity were identified
Indicator 2.1d Extent to which the key barriers to the scale-up of the MHC services were identified
Indicator 2.1e Evidence of local absorption capacity assessment performed
Indicator 2.2.f Evidence of local implementation capacity assessment performed
Judgment Criterion 2.2
The Project’s interventions selected and the implementation strategy adopted address the needs and problems identified (qualitative judgment “fully address”; “partially address” and “not address”)
Indicator 2.2a Evidence of analysis of strategic options for defining the Project’s interventions and implementation strategy performed and its quality in light of the international experience
Indicator 2.2b Evidence of justification of the recommended implementation strategy, and of management and coordination arrangements;
Indicator 2.2c Extent to which the Project interventions as planned target the leading causes of MCH mortality and morbidity
Indicator 2.2d Extent to which the Project interventions as planned target the key barriers/bottlenecks in scale up of the MCH services
Judgment Criterion 2.3
The Project interventions selected and the implementation strategy take into account the lessons learnt and recommendations derived from the implementation experience of the IMCHS I Phase (qualitative judgment: yes, partially, no)
Indicator 2.3 The share of recommendations and lessons learned from IMCHS I adopted/incorporated
Judgment Criterion 2.4
The Project interventions selected and the implementation strategy take into account the local absorption and implementation capacity (qualitative judgment: yes, partially, no)
Indicator 2.4a Evidence that local absorption capacity of the Project beneficiaries: MoH, health providers and community are taken into account during the design of the Project interventions.
Indicator 2.4b Evidence that local implementation capacity of the key Project partner - MoH are taken into account during the design of the Project interventions and implementation strategy.
Feasibility High
Methods Sub-questions:
1. Have the needs/problems originally identified changed?
2. How flexible was the Project in updating/revising the Project interventions in order to facilitate changes in circumstances?
3. If formal needs assessment was not conducted, how the needs/problems were identified?
4. If formal assessment of local absorption capacity was not performed, how the local absorption capacity was determined (if determined at all);
5. If formal assessment of local implementation capacity was not performed, how the local absorption capacity was determined (if determined at all);
41 IMCHS II Final Evaluation Report –August 31 / 2015
6. If analysis of the strategic options was not conducted, how the implementation strategy and interventions were selected?
Data collection methods:
DR; KII; Focus Group Discussions (FGD)
Information sources:
Documents
The national development documents (Poverty Reduction Strategy; National Development Strategy)
EQ3 Were all relevant stakeholders identified and involved in a balanced manner in the
Project design, implementation and monitoring, including beneficiaries?
Origin/Utility This question will help to obtain answers across several evaluation “topics” included
in the original ToR: (a) the quality of the identification of key stakeholders and target
groups (including gender analysis and analysis of vulnerable groups) and of
institutional capacity issues ; (b) stakeholder participation in the design and in the
management/implementation of the project, the level of local ownership, and issues of
absorption capacity; (c) Appropriateness of initial consultation with, and participation
by local key stakeholders and (d) whether intended beneficiaries participated in the
intervention. The findings and conclusions will be used to assess the Project
relevance. However answers to this question will also be important to establish the
national ownership and thus assess the sustainability prospects of the Project.
Judgment Criterion 3.1
All relevant stakeholders were identified using the rigorous approach: stakeholder analysis, gender analysis and analysis of the vulnerable groups (qualitative judgment: fully, partially, no)
Indicator 3.1a Evidence of stakeholder, gender and vulnerable groups analysis conducted and its correspondence to the accepted analytical approaches
Indicator 3.1b Number and share of all key stakeholders and target groups identified
Judgment Criterion 3.2
Identified key stakeholders were involved in the design of the Project (qualitative judgment: fully, partially, no)
Indicator 3.2a Evidence that the key stakeholders identified were consulted during the Project design phase (minutes of meetings, stakeholder conferences etc.)
Indicator 3.2b Number of share of the identified key stakeholders that were consulted
Judgment Criterion 3.3
Appropriate mechanisms were put in place to ensure the key stakeholder participation in the Project implementation and monitoring (qualitative judgment: yes, partially, no)
42 IMCHS II Final Evaluation Report –August 31 / 2015
Indicator 3.3a Evidence of mechanisms for stakeholder participation in the Project coordination/implementation
Indicator 3.3b Evidence of mechanisms for stakeholder participation in the Project monitoring
Judgment Criterion 3.4
There was an appropriate balance of responsibilities between the various stakeholders
during the design, implementation and monitoring of the Project (yes, no);
Indicator 3.4 The Respondents and ET judgment on the appropriate balance of responsibilities at all
stages
Feasibility High
Methods Sub-questions:
1. How the key stakeholders were identified, if the formal analytical methods were not used?
2. If there were no formal mechanisms, how (if) stakeholder participation in the Project implementation and monitoring was ensured?
3. Which accompanying measures (if any) have been taken by the MoH to ensure the
appropriate balance of responsibilities between the various stakeholders?
EQ8 Have the planned benefits been delivered and received, as perceived by all key
stakeholders (including women and men and specific vulnerable groups)?
Origin/Utility This question originated from the evaluation topic included in the ToR: “whether the
planned benefits have been delivered and received, as perceived by all key
stakeholders (including women and men and specific vulnerable groups and serves for
the assessment of the stakeholder’s perception of the Project effectiveness.
Judgment Criterion 8.1
The Project delivered the planned benefits to the key stakeholders and beneficiaries (qualitative judgment: yes, partially, no)
Indicator 8.1a Evidence of the Project benefits delivered to the MoH (all three tiers)
Indicator 8.1b Evidence of the Project benefits delivered to health providers (doctors and nurses)
Indicator 8.1c Evidence of the Project benefits delivered to patients/community (including women and
men and specific vulnerable groups)
Judgment Criterion 8.2
The key Project stakeholders and beneficiaries have a perception that they received the planned benefits (qualitative judgment: yes, partially, no)
Indicator 8.2a The Project benefits received as identified by the MoH (all three tiers)
Indicator 8.2b The Project benefits received as identified by health providers (doctors and nurses)
Indicator 8.2c The Project benefits received as identified by patients/community (including women and
men and specific vulnerable groups)
Feasibility Medium: due to the social and cultural barriers observed in the RoU it may be difficult to obtain true and sincere perceptions through the FGDs
Methods Sub-questions:
1. Did you have any specific expectations towards the potential benefits the Project may
46 IMCHS II Final Evaluation Report –August 31 / 2015
have delivered to you, if yes, have they been met (for MoH and health providers)?
48 IMCHS II Final Evaluation Report –August 31 / 2015
Other national stakeholders
EQ12 To what extent the intended results could have been achieved at a higher level of
quantity / quality by changing for example:
a) Responsiveness and flexibility of project management
b) Monitoring of risks and external factors
c) Balance of responsibilities between the various stakeholders
d) Accompanying measures taken or to be taken by the government of the RoU.
Origin/Utility This question mirrors the respective evaluation topic included in the ToR. The
responses to this question will provide key information on the underlying reasons for
the observed Project effectiveness.
Judgment Criterion 12.1
The Project could have performed better in achieving the intended results (possible, not possible)
Indicator 12.1 The ET judgment on the possibility of achieving higher level (quantity and or/quality) of the intended results
Judgment Criterion 12.2
If the plausible possibility is established, the improved results would have been achieved by (a) better responsiveness and flexibility of the Project management (yes/no); (b) improved monitoring of risks and external factors (yes/no); (c) shifting balance of
responsibilities between the various stakeholders (yes/no); (d) Accompanying
measures taken or to be taken by the government of the RoU (yes/no).
Indicator 12.2 The ET judgment on the means for achievement of better intended results
EQ14 Extent to which the costs of the Project justified by the benefits delivered whether or
not expressed in monetary terms (value for money) in comparison with similar
projects?
Origin/Utility This question is derived from the evaluation topic included in the ToR: “Extent to
which the costs of the project have been justified by the benefits whether or not
expressed in monetary terms in comparison with similar projects or known alternative
approaches, taking account of contextual differences and eliminating market
distortions. The response to this question will allow assessing the Project efficiency.
Judgment Criterion 14.1
The Project has delivered the key outputs (trained personnel) at lower or comparable costs to other similar projects in similar contexts, including the comparable outputs in IMCHS I (yes, no).
Indicator 14.1a Average costs per trainee for the Projects CD activities vs. the average costs per trainee for the similar projects
Indicator 14.1b Average costs per trainee for the Projects CD activities vs. the average costs per trainee for
50 IMCHS II Final Evaluation Report –August 31 / 2015
the IMCHS I.
Feasibility Low: the ET foresees difficulties in obtaining the comparable information from similar Project are foreseen.
IDP reports on similar projects (to be identified)
Interviewees
PT
IMPACT
EQ17 To what extent the Project’s overall objective has been achieved as intended and
whether the Project:
g. have been facilitated/constrained by external factors
h. have produced any unintended or unexpected impacts, and if so how have
these affected the overall impact.
i. have been facilitated/constrained by project/programme management, by co-
ordination arrangements, by the participation of relevant stakeholders
j. have contributed to economic and social development
k. have contributed to poverty reduction
l. have made a difference in terms of cross-cutting issues like gender equality,
environment, good governance, etc.
Origin/Utility This question assesses the ToR evaluation topics: (a) “Extent to which the objectives
of the project have been achieved as intended in particular the project planned
overall objective” and (b) “whether the effects of the project:
1. have been facilitated/constrained by external factors
2. have produced any unintended or unexpected impacts, and if so how have these
affected the overall impact.
3. have been facilitated/constrained by project/programme management, by co-
ordination arrangements, by the participation of relevant stakeholders
4. have contributed to economic and social development
5. have contributed to poverty reduction
6. have made a difference in terms of cross-cutting issues like gender equality,
environment, good governance, conflict prevention etc.
7. were spread between economic growth, salaries and wages, foreign exchange,
and budget.
The item 7 from the list above was excluded as non relevant for this Project.
The question is instrumental in establishing the Project impact.
Judgment Criterion 17.1
Positive changes in the OVIs measuring the achievement of the Project’s overall
objective were achieved (substantial positive change, moderate positive change,
incremental positive change, no change, increments negative change, moderate
negative change, substantial negative change)
Indicator 17.1a Infant mortality rate
52 IMCHS II Final Evaluation Report –August 31 / 2015
Indicator 17.1b Neonatal mortality rate
Indicator 17.1c Maternal mortality rate
Judgment Criterion 17.2
This change in OVIs can be associated with the Project’s contribution (qualitative
judgment: associated/not associated)
Indicator 17.2 The ET judgment on Project’s contribution to the observed impact
Judgment Criterion 17.3
The Project’s overall effect was affected by the: (a) the Project management
(facilitated, no effect, constrained); (b) by the coordination arrangements and
participation of the relevant stakeholders ((facilitated, no effect, constrained).
Indicator 17.3 The ET judgment on the effect of the internal factors
Judgment Criterion 17.4
The Project have contributed to economic and/or social development and poverty
reduction (likely contribution, no contribution)
Indicator 17.4a Key economic and social indicators available for the Project targeted regions
Indicator 17.4b The ET judgment on the Project’s contribution to the economic/social development
Judgment Criterion 17.5
The Project has made difference in terms of cross-cutting issues like gender equality,
environment, good governance, etc (yes/no)
Indicator 17.5 The ET judgment on the Project’s effect on cross-cutting issues
Feasibility Low to Medium, the regional data on the key socioeconomic indicators (Gini, poverty level) may not be available. The establishment of the causal links and even the likely association between the observed impact across key MCH indicators and the Project may prove difficult.
Methods Sub-questions:
None
Data collection methods:
DR; KII
Information sources:
Documents
National statistical reports by the State Committee of the Republic of Uzbekistan on Statistics
Reports of Institute of Health and Medical Statistics (IHMS) of the Ministry of Health of the Republic of Uzbekistan Estimates of the UN Inter- Agency Group for Child Mortality Estimation (UNICEF, WHO, WB, UNPD)
Estimates on Maternal Mortality developed by WHO, UNICEF, UNFPA, WB
EQ18 Whether the key national stakeholders have maintained the agreement with the Project
objectives?
Origin/Utility This question derives from the ToR evaluation topic: “the ownership of objectives and
53 IMCHS II Final Evaluation Report –August 31 / 2015
achievements, e.g. how far all stakeholders were consulted on the objectives from the
outset, and whether they agreed with them and continue to remain in agreement”. The
first part on consultation is partially covered under the EQ 3. The question is
instrumental in assessing the Project sustainability prospects.
Judgment Criterion 18.1
The key national stakeholders (MoH, regional and local authorities, academia, health providers) have been consulted in defining the Project objectives (yes, partially, no)
Indicator 18.1 Evidence of consultations with key national stakeholders on the Project objectives
Judgment Criterion 18.2
The key national stakeholders are in agreement with the Project objectives at the time of the evaluation (yes, partially, no)
Indicator 18.2 The ET judgment on the agreement of the key national stakeholders
EQ20 To what extent the Project is incorporated in the national budgets and embedded in
local institutional structures and whether the MoH and other key national stakeholders
have been properly prepared for taking over, technically, financially and managerially
and what is the likelihood that they will be capable of continuing the flow of benefits
after the project ends?
Origin/Utility This question, is formulated from the ToR evaluation topic: “institutional capacity,
e.g. of the Government (e.g. through policy and budgetary support) and counterpart
institutions; the extent to which the project is embedded in local institutional
structures; if it involved creating a new institution, how far good relations with
existing institutions have been established; whether the institution appears likely to be
capable of continuing the flow of benefits after the project ends (is it well- led, with
adequate and trained staff, sufficient budget and equipment?); whether counterparts
have been properly prepared for taking over, technically, financially and
managerially”. The topic is also partially covered under the EQ1. The EQ 20 question
is critical for assessing the Project sustainability prospects.
Judgment Criterion 20.1
The GoRU is likely to take over the key activities of the Project that require continuation, once the Project ends (most likely/likely/less likely)
Indicator 20.1a Evidence of reflection of the Project supported priorities in the national budget
Indicator 20.1b Level of funding available for take over of the Project supported activities, once the Project support ends.
Indicator 20.1c The ET judgment on the likelihood of adequate funding to become available, once the
Project support ends
Judgment Criterion 20.2
The MoH is likely to maintain and effectively operate the MCH coordination mechanism established with the Project support (Most likely, likely, less likely)
Indicator 20.2 Respondents’ and the ET judgment on the likelihood of long term maintenance of the MCH coordination mechanism.
Feasibility Medium: The detailed budget breakdown for the next three years may not be available.
Methods Sub-questions:
55 IMCHS II Final Evaluation Report –August 31 / 2015
None
Data collection methods:
DR; KII
Information sources:
Documents
Annual Budgets
Interviewees
PT
Policy Makers (GoU, MoH central, regional)
IDPs
EQ21 How adequate is the project budget for its purpose particularly phasing out prospects?
Origin/Utility This question is formulated from the ToR evaluation topic: “the adequacy of the
project budget for its purpose particularly phasing out prospects”. The topic is also
partially covered under the EQ20 and serves for assessing the Project sustainability
prospects.
Judgment Criterion 21.1
The Project will likely be able to finalize all the planned activities, including any planned handover measures (most likely/likely/less likely)
Indicator 21.1a Evidence of the handover measures in the Project budget
Indicator 21.1b The ET judgment on the likelihood of the adequate Project funding
Feasibility High:
Methods Sub-questions:
None
Data collection methods:
DR; KII
Information sources:
Documents
The Project Budget
Interviewees
PT
MoH
EQ22 Have the financial sustainability, more specifically, the financial access to the Project
supported MNCH services for the Project’s beneficiaries improved?
Origin/Utility This question is formulated from the ToR evaluation topic: “financial sustainability,
e.g. whether the products or services being provided are affordable for the intended
beneficiaries and are likely to remained so after funding will end; whether enough
funds are available to cover all costs (including recurrent costs), and continued to do
so after funding will end; and economic sustainability, i.e. how well do the benefits
(returns) compare to those on similar undertakings once market distortions are
eliminated”. The question was modified to reflect the Project specifics, e.g. it will be
very difficult, if not impossible to evaluate economic sustainability of this particular
Project. The topic is also partially covered under the EQ20 and serves for assessing
the Project sustainability prospects.
Judgment Criterion 22.1
Financial access to MNCH services at primary and secondary level has improved (improved/not improved/worsened)
Indicator 22.1a % of the population that can not afford the necessary health services
Indicator 22.1b % of private expenditures on health
Indicator 22.1c Respondents and the ET judgment on financial access to the MNCH services
Feasibility Low: time series data for the evaluated period on the financial access may not be
56 IMCHS II Final Evaluation Report –August 31 / 2015
available. FGD may not render the necessary information due to the existing cultural and social barriers.
Methods Sub-questions:
None
Data collection methods:
DR; KII
Information sources:
Documents
Health Expenditure and utilization surveys (to be identified)
National Health Accounts (to be identified)
National Statistics
IDP reports (to be identified)
Interviewees
PT
MoH
IDPs
FGD
Health providers (doctor and nurses)
Patients/community
EQ23 Whether the knowledge, services and technologies in MNCH introduced and/or
provided by the Project fits in with existing needs, culture, traditions, skills or
knowledge and how well these were absorbed by the beneficiaries (health providers
and families/communites)?
Origin/Utility This question is formulated from the ToR evaluation topic: “technical (technology)
issues, e.g. whether (i) the technology, knowledge, process or service introduced or
provided fits in with existing needs, culture, traditions, skills or knowledge; (ii)
alternative technologies are being considered, where possible; and (iii) the degree in
which the beneficiaries have been able to adapt to and maintain the technology
acquired without further assistance”. The question was modified in accordance with
the specifics of the Project. The question covers important aspects of the Project
sustainability.
Judgment Criterion 23.1
The knowledge provided through the Project supported trainings is appropriate (appropriate/less appropriate/not appropriate) and adequately absorbed by health providers (well absorbed/partially absorbed/not absorbed)
Indicator 23.1a Results of the FUAT
Indicator 23.1b Respondents and the ET judgment on the appropriateness of the provided knowledge through the Project supported trainings
Judgment Criterion 23.2
The patients/beneficiaries are satisfied with the MCNH services provided by the trained providers patients. (very satisfied, satisfied, not satisfied, not at all satisfied)
Indicator 23.2 Level of patient satisfaction with provided services
Judgment Criterion 23.3
The patients/communities have improved knowledge of MNCH issues, including the those covered under the nutrition component (yes/no)
Indicator 23.3 KAP survey indicators (to be determined) for which comparable time-series data is available
Feasibility Low to Medium: Data availability on patient/community knowledge and satisfaction levels may be limited, potential difficulties in obtaining data from the FGD
57 IMCHS II Final Evaluation Report –August 31 / 2015
Methods Sub-questions:
None
Data collection methods:
DR; KII; FGD
Information sources:
Documents
Project Documents (Inception report, progress reports, other M&E reports)
KAP survey
FUAT and JMT reports
Other studies (to be determined)
Interviewees
PT
MoH
IDPs
FGD
Health providers (doctors and nurses)
Patients/community
Mutual Reinforcement (Coherence)
EQ24 Was the Project consistent with, and supportive of, the policy and programme
framework within which the project is placed, in particular the EC’s Country Strategy
Paper and National Indicative Programme?
Origin/Utility This question is formulated from the ToR evaluation topic: (a) “Considering other
related activities undertaken by GoU or other donors, at the same level or at a higher
level: likeliness that results and impacts will mutually reinforce one another, or
likeliness that results and impacts will duplicate or conflict with one another”; and (b)
“Connection to higher level policies (coherence). Extent to which the
project/programme (its objectives, targeted beneficiaries, timing, etc.): is consistent
with and supportive of, the policy and programme framework within which the project
is placed, in particular the EC’s Country Strategy Paper and National Indicative
Programme, and the Partner Government’s development policy and sector policies.”
The last part of the topic is covered under the EQ1 and serves for assessing the
Project Coherence criteria.
Judgment Criterion 24.1
The Project objectives, targeted beneficiaries, timing is consistent with and supportive of the EC’s Regional Cooperative Programme with Central Asia, Country Strategy Paper and National Indicative Programme for the RoU (fully, partially, not consistent/supportive).
Indicator 24.1 Degree of alignment with Central Asia DCI Indicative Programme, CSP, NIP objectives and priorities
Feasibility High:
Methods Sub-questions:
1. How have the Project designers assessed the likeliness of the likeliness to reinforce / contradict other IDP and EU programmes and policy/strategy frameworks?
Data collection methods:
DR; KII
Information sources:
Documents
Project documents (DoA, Inception report)
Central Asia DCI Indicative Programme for 2011-2013,
58 IMCHS II Final Evaluation Report –August 31 / 2015
(2014-2015?)
Country Strategy Papers
National Indicative Programme
IPD Project Documents (to be determined)
Interviewees
EU Delegation
PT
MoH
IDPs
EU Added Value
EQ25 What is the EU value added of the Project?
Origin/Utility This question is formulated from the ToR evaluation topic: “Connection to
interventions of Member States. Extent to which the project/programme (its objectives,
targeted beneficiaries, timing, etc.): is complementary to the intervention of EU
Member States in the region/country/ are; is coordinated with the intervention of EU
Member States in the region/country/area; is creating actual synergy (or duplication)
with interventions of EU Member States; involves concerted efforts by EU Member
States and the EC to optimise synergies and avoid duplication.” The topic is also
partially covered under the EQ20 and serves for assessing the Project EU value
added.
Judgment Criterion 25.1
The Project is complementary and coordinated with the intervention of EU Member
States in the RoU (yes/no)
Indicator 25.1 Evidence of complementarity with the EU member country intervention
Judgment Criterion 25.2
There is a concerted efforts by EU Member States and the EC to optimise synergies
and avoid duplication
Indicator 25.2 Evidence of coordination efforts
Feasibility High:
Methods Sub-questions:
How have the Project designers assessed the likeliness to reinforce / contradict other EU member state interventions
Data collection methods:
DR; KII
Information sources:
Documents
Project documents (DoA, Inception report)
Other EU member project documents (to be determined)
Interviewees
EU Delegation
PT
MoH
IDPs representing the EU member states
59 IMCHS II Final Evaluation Report –August 31 / 2015
Annex 4. List of Documents and Literature Reviewed
Improvement of Mother and Child Health Services in Uzbekistan Phase II Project Documents
1. Action Fiche #3 2. Financial and Administrative Framework Agreement between the EC and UNICEF 3. Contract # 2012/295-374 – Description of Action, Annexes and Budget 4. Contract Modification December 2013 5. Inception Report 6. First Interim Report (2012-2013) 7. Second Interim Report (2013-2014) 8. First Interim Report (2012-2013) 9. Periodic Updates (total 24 documents – from 2012 through 2015 10. Mid-term Internal Evaluation Report 2014 11. Nutrition BCC Materials 12. ROM Monitoring Report 2013 13. Joint Monitoring Visit Reports (5 reports, 2013-2015) 14. Baseline Survey Report 15. Final draft COMMUNITY MOBILIZATION GUIDELINES, UNICEF 16. HFA (Quality of Health Care) LQAS Survey Report 17. Training Assessment Tools and Training Modules 18. Quality Assessment Tool for Paediatric Hospitals, WHO 19. Quality Assessment Tool for Perinatal Centers and Maternities, WHO/UNICEF
Supply of equipment to the IMCHS II 20. Documentation for the Medical Equipment Supply Contract (ToR for the international
consultant, equipment specifications, contract documents) EU Framework Documents
21. Central Asia DCI Indicative Programme 2011-2013 22. Multiannual indicative programme. Regional Central Asia 2014-2020 23. Multiannual indicative programme. Uzbekistan 2014-2020 24. Regional Strategy Paper for Assistance to Central Asia for the period 2007-2013 25. Partnership and Cooperation Agreement between the EC and Uzbekistan (PCA) –
1996 26. EU and Central Asia strategy for a new partnership 2007
Other Documents
27. Mid-Term Review Report UNICEF Uzbekistan. 2013 28. Making Pregnancy Safer/Promoting Effective Perinatal Care/Essential Obstetric and
Neonatal Care Training Course. UNCIDF 2013 29. Progress Toward the Millennium Development Goals and Other Measures of the well-
being of Children and women . UNICEF. 2012 30. Formative Evaluation of Improvement of Mother and Child Health Services in
Uzbekistan (Phase I), Final Report 2011 31. Health in Transition. Uzbekistan. WHO, 2007, 2013 32. Multiple Indicator Cluster Survey 2006 33. Presidential Decree 2221 from 1 August 2014 “On approval of the 34. the President’s Resolution N1096 from 13 April 2009 “On Additional Measures for
Protection of Maternal and Child Health and Formation of the Healthy Generation” 35. the President’s Resolution N2221 from 1 August 2014 “On the State Program for
Further Improvement of the Reproductive Health of the population and the Protection of Maternal, Child and Adolescent Health in Uzbekistan for 2014-2018”.
60 IMCHS II Final Evaluation Report –August 31 / 2015
Annex 5 – List of Persons/organizations consulted
8 June – Monday
European Union Delegation Office
Mr. Doniyor Kuchkarov, Project Manager
9 June – Tuesday
Center of Development of Medical Education
Ms. Alimova M.Kh. - Director of the Center of Development of Medical Education
Course Director: “Training of pedagogical personnel of medical institutes and nurses schools”
Institute for Postgraduate Medical Education
Mr. Asadov Damin Abdurahimovich - Head of the Department at the Institute for Postgraduate
Medical Education
Director of the Health Management Course
Republican Perinatal Care Center
Ms. Lubchich A.S. - Director of the Republican Perinatal Care Center
Course Director: “Perinatal Care”
Ministry of Health, Republic of Uzbekistan
Mr. Laziz N Tuychiev - Deputy Minister of Health
Ms. Nodira Islamova – Head of Maternity, Mother and Child Health Department
Ms. Hasanova Dilfuza – Deputy Head, Mother and Child Health Department
Mr. Khamraev Naim - Specialist, Department of International Relations
Ms. Diyora Arifdjanova – National Coordinator of the project
10 June – Wednesday
UNICEF Uzbekistan Country Office
Mr. Robert Fuderich - UNICEF Representative in Uzbekistan
Ms. Berina Arslanagic-Ibisevic – UNICEF Deputy Representative in Uzbekistan
Ms. Swetlana Stefanet – Chief of Health, UNICEF Uzbekistan Country Office
Mr. Maxim Fazlitdinov - Communication for Development Officer, UNICEF Uzbekistan Country
Office
Mr. Fakhriddin Nizamov – Health Officer, UNICEF Uzbekistan Country Office
"Improvement of Mother and Child Health Services in Uzbekistan, phase II" Project office
Ms. Nargiz Shamilova - Team Leader of the "Improvement of Mother and Child Health Services in
Uzbekistan, Phase II" project
11 June – Thursday
UNFPA country office
Mr. Karl Kulessa - Country Representatives
Mr. Fuad Aliev - Assistant representative
Perinatal Care Center of Tashkent City
Mr. Kasimov Atham, Deputy Director of the Perinatal Care Center of Tashkent City
Ms. Usmanova M.S., Deputy Director of the Perinatal Care Center of Tashkent City
Course Director: “Neonatal resuscitation, essential newborn care, neonatal equipment”
WHO Country Office
Mr. Asmus Hammerich, WHO Representative
Ms. Zulfia Atadjanova, National Professional Officer
Republican Specialized Scientific and Practical Medical Center of Pediatrics
Ms. Akhmedova D.I., Director of the Republican Specialized Scientific and Practical Medical Center
of Pediatrics
Course Director: “Integrated management of childhood illnesses at community, primary care, and
hospital level. Community based BCC activities.”
Ms. Salikhova K.S., Deputy Director, Republican Specialized Scientific and Practical Medical Center
of Pediatrics
Course Director: “Breast feeding and baby friendly hospital initiatives”
61 IMCHS II Final Evaluation Report –August 31 / 2015
12 June - Friday
National Coordinating Unit of the European Union’s Technical assistance Programme
Mr. Jamshed Sh. Sharapov - Director of the National Coordinating Unit
Ms. Nargiza M. Samandarova – Senior Expert of the National Coordinating Unit
Medconcept Europe
Mr. Timur Minibaev – Commercial Director of Medconcept Europe in Uzbekistan
European Union Delegation Office
Ms. Assunta Testa – Programme Manager of the EUD in Uzbekistan
UNICEF Uzbekistan Country Office
Ms. Marina Andranopoli - International Nutrition Specialist of the Project
15 June - Monday
Khorezm Oblast Health Department
Mr. Kudrat Jumaniayazov - Deputy Head (on maternal and child health) of the Khorezm Oblast Health
Department
Primary Health Care Centre (Rural Medical Point) “Chotkupir”, Urgech District, Khorezm Oblast
Ms. Ruzmetova Myassar, Chief of the Primary Health Centre (rural) “Chotkupir”, Urgech District,
Khorezm Oblast
Paediatric Department of the Khanka Regional Hospital, Khanka District, Khorezm Oblast
Ms. Nazokhat Khamraeva, Chef of the Paediatric Department of the Khanka Regional Hospital,
Khanka District, Khorezm Oblast
Mr. Bekjanov Jumanazar, Chief of the Khanka Regional Hospital, Khanka District, Khorezm Oblast
Maternity Hospital of the Boghat District, Khorezm Oblast
Mr. Muradov Dilmurod - Chief of the Maternity Hospital of the Boghat District, Khorezm Oblast
Khorezm Oblast Perinatal Centre, Urgench City, Khorezm Oblast
Ms. Gulsara Kurbanbaeva – Head of the Khorezm Oblast Perinatal Centre, Urgench, Khorezm Oblast
16 June - Tuesday
Ministry of Health of the Republic of Karakalpakstan
Mr. Hodjiev Daniyar Shamuratovich - The Minister of Health of the Republic of Karakalpakstan
Mr. Kural Kamalov - First Deputy Minister
Ms. Dametken Bekbaeva - Deputy Minister in charge of MCH
of Women commeettes from Konshi, Orjap, and Karakol Mahalays.
17 June - Wednesday
World Bank
Mr. Iqboljon Ahadjonov - Health specialist of the World Bank
European Union Delegation Office
Ms Andrea Raith - Deputy Head of Finance Section,\
Ms. Andrea Raith - Deputy Head of Finance Contract and Audit Section, European Union Delegation
Office in Uzbekistan
Ms. Leyla Abdrashitova – Finance and contract officer, European Union Delegation Office in
Uzbekistan
62 IMCHS II Final Evaluation Report –August 31 / 2015
KfW Development Bank
Mr. Shakhrukh Irnazarov – Head of KfW Office, KfW Development Bank
Ms. Nigora Karabaeva – National project Coordinator, Reproductive Health/Improvement of Mother
and Child Care, GFA Consulting Group
European Union Delegation Office
Mr. Dominique Wauters - Head of Cooperation section, European Union Delegation Office in
Uzbekistan
"Improvement of Mother and Child Health Services in Uzbekistan, phase II" Project office
Mr. Nasir Abdulaev – Monitoring and Evaluation Consultant of the project
18 June - Thursday
GIZ office in Uzbekistan
Mr. Gunnar Strote - Head of the Project, Advance Medical Technology in Uzbekistan
Ms. Nidora Murotova - Project National Coordinator, Advance Medical Technology in Uzbekistan
Ms. Raushan Ataniyazova - Team Leader, Regional programme “Health in Central Asia - Uzbekistan”
Ms. Umida Dusmatova, project assistant
European Union Delegation Office
Debriefing with the EU Delegation
Mr. Dominique Wauters - Head of Cooperation section, European Union Delegation Office in
Uzbekistan
Mr. Doniyor Kuchkarov, Project Manager
19 June - Friday
"Improvement of Mother and Child Health Services in Uzbekistan, phase II" Project office
Presentation of preliminary findings
63 IMCHS II Final Evaluation Report –August 31 / 2015
Annex 6. Distribution of Responsibilities and CVs of the Evaluation Team
Expert 1 Public Health expert -Team Leader: Akaki Zoidze
Dr Akaki Zoidze has more than sixteen years of consulting experience with the World Bank, UN agencies, EU, USAID, DFID and other international organizations, which allowed him to enrich the knowledge of health systems and health reforms and contribute to health policy development, health financing, public health and health service delivery reforms in 15 transition and developing countries: Armenia, Albania, Azerbaijan, Bosnia & Herzegovina, Georgia, Kosovo, Kyrgyz Republic, Kazakhstan, Moldova, Russia, Somalia, Tajikistan, Ukraine and Uzbekistan; These includes two long term assignments (Azerbaijan/USAID/Primary Health Care Reform Project in 2005-2007 and Georgia/USAID Health Financing Reform Project in 2004-2005) in implementing health care reform projects in the CIS Countries – Azerbaijan and Georgia. He has a considerable experience in programme monitoring and evaluation, programme identification/formulation, including the participation in 7 programme identification/formulation missions for various development partners (Azerbaijan/IFRC in 1993, Azerbaijan/IMC in 1999, Azerbaijan/DTRA in 2011, Kyrgyz Republic/Kazakhstan/WHO in 2005, Uzbekistan/UNICEF/EU in 2008, Tajikistan/EU 2008 and 2013, Ukraine 2012), 5 review and evaluation missions (Bosnia Herzegovina/GAVI 2014, Multi-Country/UNICEF 2014, Kosovo/The World Bank in 2000, Kyrgyz Republic/DFID in 2001, Georgia/EU 2008, Ukraine/UNICEF 2014, Multi-country/UNICEF); Health System Performance Assessment Programmes in Georgia, Azerbaijan, Turkey funded by various development partners.
Akaki possesses a diversified experience in evaluation of projects supporting the Mother and Child health care area: Maternal and Child Health Situation Analysis for Georgia (UNICEF in 1999); Development of Inception Report for EC funded Project “Improving the Maternal and Child Health Services in Uzbekistan (UNICEF/EU 2008); Mid Term Review of the Health Communication and Promotion for Reproductive, Maternal and Child Health Services in Ukraine (UNICEF 2014); Evaluation of UNICEF's contribution in five CEE/CIS countries (Kazakhstan, Kyrgyzstan, Moldova, Serbia and Uzbekistan) towards the reduction of and closing the inequity gap in under-5 and infant mortality and morbidity (UNICEF, 2015). In overall, Akaki Zoidze has adequate qualifications necessary for conducting this work and he is well placed to lead the team. Therefore, he will assume primary responsibility for evaluation design, planning, data analysis and report production.
Expert 2 Nutrition Specialist: Annemarie van Hoogendoorn
POSITION NAME RESPONSIBILITIES
Expert 1 - Team Leader & Public Health Expert
AKAKI ZOIDZE, MD Evaluation Team Leader was responsible for overall design, planning, implementation and preparation of the inception, the field, and the final evaluation reports. Served as a Public Health Expert on the Evaluation team. Took the lead in formulation of evaluation questions, methodology, analysis strategy and evaluation tools as well as the format of the final deliverables, performed a desk review, data triangulation and produced required inputs for the final report.
Expert 2 - Nutrition Expert
ANNEMARIE HOOGENDOORN
Contributed towards formulation of evaluation questions, analysis strategy and evaluation tools as well as the format of the final deliverables, performed a desk review, data collection and triangulation with the focus on the nutrition interventions and produced required inputs for the inception, the field and the final reports.
Expert 3 - Health Capacity Development Expert
GIORGI PKHAKADZE Contributed towards formulation of evaluation questions, analysis strategy and evaluation tools. Performed a desk review, data collection and triangulation with the focus on the health capacity development interventions and produces required inputs for the inception, the field and the final report.
64 IMCHS II Final Evaluation Report –August 31 / 2015
Annemarie Hoogendoorn is a senior Public Health Nutrition expert who holds an additional MSc in Strategic Management. She has 25 five years of experience in the area of international nutrition (in particular for young children and pregnant and lactating women) in development and emergency settings. This includes over ten years of work as short-term consultant for the EC and UN on evaluations and technical backstopping missions. She is fully committed to preventive BCC approaches for improving nutrition security, both through the health sector and in the form of other approaches including community-based programmes, and is up-to-date on the 2025 WHA nutrition targets, the SUN Movement and its 1000 days approach, and the need for complementary nutrition-specific and nutrition-sensitive interventions. Through her work, she acquired a deep knowledge of and experience with EU procedures and aid modalities, including the 2014 EU Action Plan on Nutrition.
Mrs Hoogendoorn has varied geographical experience, including some missions on nutrition in Central Asian countries (CIS) and is fully fluent in English and with good communication skills.
Expert 3 Expert in Health Capacity Development: Giorgi Pkhakadze
Dr Giorgi Pkhakadze is an international expert with 18 years working experience in technical support to the design, establishment, management, assessment, monitoring and evaluation of health and social policies and programmes including health care reform. Dr Pkhakadze holds Master/PhD degrees in Public Health.
He has a long experience with curricula and training /re-training of health care professionals, in his current function of Professor and Head of the School of Public Health (D. Tvildiani Medical University), as well as in previous projects in India and Georgia and global (including Uzbekistan, Central Asia, Eastern Europe). As a regional adviser of the UNFPA he covered 20 countries in Eastern/Central Europe and Central Asia and evaluated numerous country and regional sexual and reproductive health projects.
Dr Pkhakadze has strong analytical, report writing and presentation skills.
65 IMCHS II Final Evaluation Report –August 31 / 2015
Annex 7. Project Stakeholders
Partners Role Activities Geographical Location
NATIONAL
MoH, National Medical Institutes and Regional Health Departments
Key implementing partners. Mother and Child Health Department serves as a secretariat for MCHCC. Other relevant departments of MoH within the MoH contribute in different ways and levels of responsibility.
All Project Activities
Nation wide
Mahallas and Women’s Committees
Implementing partners at local level in community based activities (PLA)
Participatory Learning Approach
Nation wide
UNICEF Primary implementing partner provides technical support to the Project. UNICEF is complementing healthcare support through its regular programme in six additional regions; implements effective perinatal care (EPC) training and integrated monitoring of the implementation of ILBD nationwide
All Project Activities
Nation Wide
WHO Key implementing partner - provides technical input on quality assurance/certification, supportive supervision and monitoring; participates in the Joint Monitoring Team (JMT), together with MoH, and UNICEF
Technical assistance to MoH and UNICEF Effective Perinatal Care – Making pregnancy safer. IMCI, MCH and Reproductive Health. HIV/AIDS - PMTCT
Nation wide
UNFPA Supports the safe motherhood and effective perinatal care (EPC) programme by providing technical input and logistic support.
Improvement of maternal health care. Emergency Obstetric Care – Capacity building and logistic support.
Nationwide
Andijan & Djizzak
The World Bank Supports General practitioners and nurses in Primary
Infrastructure development of PHC facilities, district
Nationwide
66 IMCHS II Final Evaluation Report –August 31 / 2015
Healthcare. The World Bank Health 3 Project has been recently approved which is focused on secondary health care at the district level.
health systems (hospitals) Supply of equipment to PHC level. Quality improvement and monitoring of PHC facilities. Training of General Practitioners. Health care financing and management system.
GIZ Focuses on the improvement of the quality of care, certification and nutrition intervention in two Regions
Quality of Care
Training of HPs
Standardization
Certification and Licensing
Andijan, RK
KfW Provision of equipment for Paediatric facilities, training of HP in the use of the provided equipment
Equipment for Paediatric facilities
Nation wide
67 IMCHS II Final Evaluation Report –August 31 / 2015
Annex 8: Adoption of IMCHS I recommendations in the design of the IMCHS II programme
Responsibilities Priority/Time Frame Adoption Status for IMCHS II
Recommendation GOU UNICEF
Enhancement of the MOH Evidence Based Policy Formulation and Coordination Functions
Development of an effective donor coordination framework
The MOH lacks a good framework for Donor coordination and with the approved MCH Sector Strategy (2009-2013) the opportunity emerges to use the strategy document as a framework for streamlined donor support. If accepted, Donor assistance has to be mapped onto strategy priorities and Donor projects have to be coordinated in a manner so as to achieve strategy objectives. This would also permit effective monitoring and assessment of Donor support towards achieving the strategy objectives. The Donor Mapping Exercise is a resource management tool, consisting of two components: a) an online database which enumerates the activities of Donors, funding resources and b) an analytical report reviewing and the contribution of these activities to the overall MCH Strategy. This will allow the MOH to have: a) based on the updated MCH strategic implementation plan mapped all required inputs, resources, timeframe and monitoring and evaluation framework b) ongoing donor and state supported activities; c) gaps for resource mobilization and d) by monitoring the performance indicators be able to identify problems, discuss them and collectively define measures for further interventions.
Project can play a pivotal role in further enhancement of the coordination mechanism through provision of technical and financial assistance.
High
Partially Adopted. The Project provided continuous technical assistance to the MCH Coordination Committee (MCHCC), which was transformed from the Project Steering Committee, into a functional donor coordination mechanism during the IMCHS II implementation. The Project also assisted the GoU in the design and adoption of the MCH strategy. The evidence of existence of the online donor database was not found
68 IMCHS II Final Evaluation Report –August 31 / 2015
Establishment of Strong Secretariat
While the MCH strategy could serve as a framework for coordination, a strong “secretariat” housed within MoH will be required for donor coordination. The secretariat should have clear policies, procedures and powers to assure adequate assistance for the MOH in coordinating Donor efforts. It is doubtful that without Donors' financial and technical assistance, and willingness, such a secretariat to emerge. Therefore the MOH has to strive to mobilize Donor support for the establishment of this secretariat and its empowerment. Donor assistance will also be essential to attract the human resources required for this unit
Project can consider provision of technical and financial assistance for the formation of the secretariat as well as development of the policies and procedures.
High/Short Term
Fully Adopted. The Secretariat was established and supported by the IMCHS II
Provision of Technical Expertise and support to coordination function
Contracting an International Advisor to ensure the provision of day-to-day advisory services, as well as, mentorship, on-the-job training and guidance to the chairman, TWG, secretariat and other key staff of the Ministry.
Provide financial support. In the case of resource constraints, an alternative option could be the use of bilateral Donors on a rotation basis, who would play this role and assist the committee in setting an agenda, monitoring various project performances, facilitating discussions and formulating practical and implementable decisions.
High/ Short Term
Partially Adopted. The special international advisor was not hired, but the IMCHS II Project Team has assisted the MCHCC in coordination function
Institutionalize Joint program reviews
Institutionalization of Joint program reviews would be fundamental to ensure increased ownership by the MOH of Donor financed projects. Specifically, the enhancement of MOH leadership and program performance assessment capabilities and the improvement of the harmonization of public and Donor financed activities. For this purpose it is recommended that the MOH signs the Memorandum of Understanding (MOU) with Donors active in the sector, develops and carries out annual joint program review plans and presents the
Capitalize on experience generated from the first phase and provide technical assistance to the MOH for the development of the joint evaluation procedures and evaluation protocols.
High/ Medium Term
Partially Adopted. The Joint Annual Reviews were not institutionalized as recommended. However, the Joint Monitoring System (JMS) established for the IMCHS II partially addressed the objectives of the Joint Annual Reviews.
69 IMCHS II Final Evaluation Report –August 31 / 2015
Responsibilities Priority/ Time Frame Adoption Status for IMCHS II
Recommendation GOU UNICEF
Harmonization of legislation for effective implementation of MCH Reform
Analysis and Harmonization of MCH related legislation
Conduct analysis of the MOH legal documents and ensure harmonization. The MOH may consider to establish the legal working groups for this purpose
Consider provision of financial and technical support
High/ Short to Medium Term
Fully Adopted. The review of the MCH legislation was one of the activities under the result 1 of the IMCHS II
Elaboration and Implementation of the MCH Workforce Development Strategy
Recommendation GOU UNICEF
Development of the MCH Workforce Strategy
Mobilize donor support to support MCH workforce strategy development which addresses: a) workforce planning, b) reforms required for pre-service and post diploma education for health managers, physicians, GPs, midwives and nurses; c) continuous professional development system design, d) certification, e) remuneration and incentives
NA High/ Medium Term
Not Adopted. The IMCHS did not plan for the development of the MCH Workforce Strategy
Integration of the MCH package into pre service, post diploma education as well as continuous professional education systems
Formulate and approve detailed implementation plan and budget for integration of the MCH package into the pre service, postgraduate and continuous education systems. Regularly monitor implementation progress
Pre-service Education System: Support integration of the MCH package in the pre-service education curricula for physicians, nurses and mid-wives
Postgraduate Education
Support integration of the MCH package in the post graduate education curricula for physicians Continuous Professional Education
Support Institutional design and development of the continuous professional education system
Medium/Medium Term
Fully Adopted. IMCHS II planned and provided support for integration of MCH package into pre-service and post diploma education.
findings at the Donor coordination meetings.
70 IMCHS II Final Evaluation Report –August 31 / 2015
(CPES).
Refinement of the cascade training approach by integration into the continuous professional education system.
Development of guidelines on the selection of trainees at all levels of service provision, geographical location and facilities
Support the MOH in the development of the country wide training plan
Elaborate and Introduce cost-sharing arrangements ( trainee/CPES/Project)
Reform Licensing and Certification systems
The Government is recommended to Integrate the MCH service performance indicators in the licensing system of private MCH health service providers. The current certification system measures only theoretical knowledge of physicians, while FUAT and IM indicate low performance levels. The MOH is recommended to reform existing certification system by adding performance measurements to be generated by the national health statics system.
Jointly with WHO and international experts assist the government in reforming the licensing and accreditations systems.
Medium/ Long Term
Fully Adopted. IMCHS II planned and provided support to reform the licensing and certification systems.
Enhancement of the MCH Service Quality Assurance Function
Responsibilities Priority Adoption Status for IMCHS II
Recommendation GOU UNICEF
Modernization of existing M&E system through consolidation into the internal and external MCH service quality audit systems
Refinement of design of the National, Oblast, District and facility level MCH service audit system.
Development of the internal and external audit functions, human resource requirement, procedures, protocols, and reporting forms and
Jointly with WHO provide technical assistance to the MOH in design of the MCH service quality audit system.
Support capacity building activities of
High/ Short to Medium Term
Fully Adopted. Under the Result 1, the IMCHS II planned several activities to accommodate the IMCHS I evaluation recommendations on internal and external MCH service
71 IMCHS II Final Evaluation Report –August 31 / 2015
follow up action plan sample.
Ensure adequate funding of the audit system.
Development of the legislative base for effective implementation of the audit system
Introduction of the facility and medical personnel’s’ performance benchmarking on the facility, District, Oblast and National level
Support capacity building of the National, Oblast, District and Facility Health Managers in application of the new MCH service performance audit system
Development and dissemination of the annual reports
the National, Oblast, District and Facility Health Managers in application of the new MCH service performance audit system
Biannually analyze MCH facility performance benchmarking reports from project target oblasts, identify problems and plan corrective measures.
Support dissemination of the annual MCH service quality audit reports.
quality audit systems (development of the concept and tools for service quality audit) that also included the refinement of the Supportive Supervision System in cooperation with WHO.
Strengthening of the MCH Health Information System
Responsibilities Priority Adoption Status for IMCHS II
Recommendation GOU UNICEF
Institutionalization of the ILBD and assurance of accurate reporting of all MCH indicators through institutionalization of the Data Quality Audit (DQA) System
Monitor ILBD integration into the national health statistics reporting system and apply corrective measures when required.
Revise routing reporting forms by integration of selected MCH indicators
Design, approve and implement DQA system
Provide technical assistance to the MOH for:
a) Iintegration of selected
MCH indicators into the
routine reporting forms;
b) Building DQA capacity at
national and local levels
through provision of
support in design of the
DQA system procedures
and protocols for all MCH
related as well as training
of relevant staff
High/ Short Term
Partially Adopted. The IMCHS II planned the support to the implementation of the ILBD, however no distinct activities were included to support the institutionalization of the DQA system.
Build national and local Design and placement of the analytical function within the MOH structure;
Support the MOH in the development of the national and
Medium/ Medium to Long Term
Fully Adopted. Respective activities were planned under the
72 IMCHS II Final Evaluation Report –August 31 / 2015
capacity in MCH data analysis Build national and local MCH analytical capacity through provision of customized training on data analysis, report writing, presentation skills, preparation of the Policy briefs and papers, etc. Organize annual conferences on the findings of the MCH data analysis and proposed future actions
local MCH data analysis capacity through provision of technical assistance as well as training of respective staff on both levels.
Improve partnership with Institute of Health and Medical Statistics
Support in organization of the annual MCH conferences.
IMCHS II’s Result 1.
Strengthening of National Information, Education and Communication Function
Responsibilities Priority Adoption Status for IMCHS II
Recommendation GOU IMCHS Project
Support effective IEC activities Develop and approve National MCH sector communication strategy and implementation plan. Establish TWG represented by key experts and international partners As part of the National MCH sector communication strategy support the introduction of the peer to peer education system on the community level (Mother to mother) and using the influence of the “mother in law institute” in favor of MCH related IEC as well as community leaders (mothers/mother in laws)
Support development of the new MCH communication strategy through provision of the technical assistance
Support the implementation of MOH “peer to peer” initiative in the project target regions
High/Medium Term
Partially Adopted. The IMCHS II
planned activities under the Result
2 to support the piloting of the
Participatory Learning and Action
approach in cooperation with
Mahallas and Women’s
Committees
Expand partnership to other public institutions
Expand partnership to other public institutions (Ministry of Communication, Ministry of Education, etc) in support of the public information, education and communication activities by establishment of the inter-ministerial working group.
Sign memorandum of understanding with lead public institutions and partners on implementation of the national MCH
Assist the MOH in the design of the inter-ministerial working group’s scope of work, in design of the annual implementation plans and budgets for the national MCH communication strategy
Provide financial support for implementation of selected
High/Short Term
Not Adopted. No activities were planned in this direction under the IMCHS
73 IMCHS II Final Evaluation Report –August 31 / 2015
communication strategy. intervention
Establish close collaboration and partnership between international partners and the Institute of Health and Medical Statistics
MOH to issue a decree that regulates cooperation and collaboration forms between Institute of Health and Medical Statistics and international partners. Memorandum of Understanding is signed between the Institute of Health and Medical Statists and international partners active in health sector (with particular emphasis on MCH sector) Develop and approve integrated MCH communication annual implementation plan and budget covering all IMCHS planned activities and ensure MOH cost-sharing arrangements
Assist the MOH in formulation of the scope of cooperation and collaboration between Institute of Health and Medical Statistics and international partners
Jointly with the Institute of Health and Medical Statistics develop integrated annual MCH communication implementation plan and budget with cost-sharing arrangements.
High/Short Term
Partially Adopted. The Institute of Health and Medical Statistics was involved as a key collaborative partner for the implementation of the community based activities under the IMCHS II Result 2, however no support was provided for the establishment of the partnership for elaboration and implementation of the integrated annual MCH communication plan.
74 IMCHS II Final Evaluation Report –August 31 / 2015
Annex 9. Status of IMCHS II assumptions and risks
Overall
objective
Intervention logic External conditions and risks Status of Assumptions (external conditions) and risks
Contribute to human development in
Uzbekistan with special emphasis on
reducing child mortality rates (MDG
4) and improving maternal health
(MDG 5).
External conditions: The GoU
should support the MCH care
sector reform. The budget
allocation to the MCH care sector
should steadily increase in real
terms. The MoH should aim to
improve provision of MCH care
services at all levels.
Risks: The GoU and MoH may
pursue other political priorities
and operational goals.
Assumption remained valid and was realized. The GoU continues to
support the MCH care sector reform (as demonstrated by the presidential
decrees and dedicating the years 2012-2013-2014 to maternal child health
issues. According to MoH the budget for MCH services was increasing
over the time and MoH remains dedicated to the changes at all levels.
Risk remained valid, however have not realized yet.
Specific
objective
Intervention logic External conditions and risks Status of Assumptions (external conditions) and risks
Support the MoH to implement MCH
care sector reforms, increase the
quality of MCH care services, and
develop the capacity of families to
adopt healthy behaviors.
External conditions: The MoH
should have a clear vision of the
MCH care sector reform,
proactively lead the reform
process, and act as a full capacity
partner in project
implementation.
Risks: The MoH will limit its role
to reacting to requests of GoU,
UNICEF, WHO, and other IDP.
Assumption remained valid and was realized. The MoH had a clear
vision of the MCH care sector reform as articulated in the Presidential
Decree N2221. MCHCC was proactively led by the MoH, who also acted
as a full capacity partner in project implementation.
Risk remained valid, however was not realized.
Expected results Intervention logic External conditions and risks Status of Assumptions (external conditions) and risks
Result 1. Institutional strengthening. The MoH should be willing to
integrate the result based
Assumption remains valid and has yet to be fully realized. While MoH
appears to be willing to integrate the results based management approach
75 IMCHS II Final Evaluation Report –August 31 / 2015
Skills and operational capacity of the
MoH is enhanced to effectively
contribute to the process of reform of
the MCH care system.
management approach in working
processes at all levels and should
undertake sustainable efforts to
do so.
in working processes at all levels, evidence of the sustainable efforts to do
so has yet to be demonstrated
No risks were identified.
Expected results Intervention logic External conditions and risks Status of Assumptions (external conditions) and risks
Result 2. Capacity development and
empowerment. The capacity of health
workers to provide quality health
services is improved. The capacity of
families to adopt healthy behaviors
and demand better health services is
enhanced.
The MoH should be willing to
change the practices of MCH care
system managers and health
professionals according to
UNICEF and MoH
recommendations.
The MoH should be willing to
implement the PLA approach in
community work.
The patronage nurses, Mahalla
Advisors, and community
members should perceive the
proposed community
interventions as practical
solutions to local problems and
actively support them.
Assumption remains valid and was fully realized. MoH has indeed shown
interest to work at community-level and is in the process of allocating
budget support for improvement of the nutritional status among the
population in Uzbekistan. Women in pilot Mahallas have shown interest to
adopt better child feeding and child care practices.
Risk remained valid, however not realized. The budgetary risks mentioned
in the Identification Fiche did not occur as GoU allocated sufficient
budget to increase the number of Patronage Nurses and also is preparing
a Strategy (with budget) on nutrition improvement
Key activities and sequence Pre-conditions required to implement activities Status of Assumptions (external conditions) and risks
i) 1.1. Support the MoH to
establish and operate the
MCHCC, its secretariat, and
JMT.
The MoH should be willing to assume the leading role in MCH care
sector coordination and accept monitoring of MCH sector by the
JMT.
Assumption remains valid and was fully realized. MoH has
assumed a leading role in MHC care sector coordination
ii) 1.2. Establish a formal
certification system and
The administration of MCH should be willing to implement changes,
required by the certification standards and must have the required
Assumption remains valid and has yet to be fully realized. While
MCH administration at national level appears to be willing to
76 IMCHS II Final Evaluation Report –August 31 / 2015
process for the MCH care
institutions.
technical capacity and financial resources to implement them. The
proposed certification standards should be acceptable and feasible
for implementation. The legislative base should be in place.
implement changes, according to key informants, the required
technical capacity and financial resources to implement them are
still lacking at oblast and local levels.
iii) 1.3. Elaborate and pilot quality
improvement mechanisms for
the MCH care institutions.
The administration of MCH should be willing to implement changes,
required by the quality improvement procedures and should have the
required technical capacity and financial resources to do so.
The proposed quality improvement mechanisms should be feasible
for implementation.
Assumption remains valid and has yet to be fully realized. While
MCH administration at national level appears to be willing to
implement changes, according to key informants, the required
technical capacity and financial resources to implement them are
still lacking at oblast and local levels.
1.4. Pilot the Health System
Strengthening activities in 2
regio
The WHO proposals should be politically acceptable for the MoH
and feasible for implementation.
Assumption remains valid and was fully realized. WHO proposals
for improvement of children rights was politically acceptable and
feasible
1.5. Perform need assessment,
elaborate specification of
essential medical equipment
for provision of quality EPC,
NR&ENBC services
MoH should improve the equipment maintenance system Assumption remains valid and has yet to be fully realized. No
evidence for changes in the MoH equipment maintenance system
were identified by the ET.
Key activities and sequence Pre-conditions required to implement activities Progress towards logical framework indicators,
as of July 2014
2.1.1. Conduct training and
supervision of healthcare
providers on maternal,
newborn and child survival
packages as per the approved
standard protocols and
guidelines.
Team of course directors should be established. The training centers
should be renovated and equipped. The training materials should be
updated, approved by the MoH, and a sufficient number of copies
should be printed. MoH should issue an order on organization of
training activities. The recipients of funds should be trained to use the
bank transfer instrument for implementation of activities and provide
the necessary documents for timely closure of DCT.
Assumption remained valid and fully realized. All the relevant pre-
conditions were met.
2.1.2. Support the MoH to
implement the newborn
survival, child survival, and
nutrition modules in
graduate and postgraduate
curricula for MCH care
The MoE and MoH should reach an agreement on the necessity to
implement the modules in the graduate and postgraduate curricula for
MCH care professionals.
The course directors should adapt the modules to the MCH care
policies of the GoU, MoH, and local conditions of Uzbekistan.
Assumption remained valid and fully realized. All the relevant pre-
conditions were met.
77 IMCHS II Final Evaluation Report –August 31 / 2015
professionals.
2.1.3. Train health care
system managers to support
improvement of MCH care
services.
The training materials should be updated, approved by the MoH, and a
sufficient number of copies should be printed. The MoH should issue
an order on organization of training activities.
Assumption remained valid and fully realized. All the relevant pre-
conditions were met.
2.1.4. Establish a platform
for exchange of experience,
discussion and dissemination
of best practices on MCH
sector reform issues.
MCHCC should be established and become fully functional.
MoH should agree to monitor MCH sector by UNICEF & WHO
The MoH should agree to lead the process of discussions and
exchange of experience on MCH care sector reform issues.
Assumption remained valid and fully realized. All the relevant pre-
conditions were met.
Key activities and sequence Pre-conditions required to implement activities Progress towards logical framework indicators,
as of July 2014
2.2.1. Establish community
based behavior change
mechanism based on
participatory learning action
approach (PLA) under the
ownership of selected
Mahallas and primary
healthcare institutions
(SVP).
MoH should accept the principles of PLA and agree to pilot it in
selected Mahallas.
Availability of KAP study results
The population of selected Mahallas should accept the principles of
PLA and should be willing and capable to support the proposed
activities.
The project should be well documented and plans for project
improvement and scale up envisaged under MoH leadership
Assumptions remained valid and fully realized. All the relevant pre-
conditions were met. However, the KAP study results became
available at a later then expected stage of the Project
implementation.
2.2.2. Develop PLA
modules, BCC materials and
tools which promote healthy
behaviors, including
nutrition alongside national
awarenes visibility activities.
The topics of PLA modules, BCC materials and tools should be based
on a sound knowledge of real health priorities and modern culture of
Mahallas.
Availability of the results of KAP study
Assumptions remained valid and fully realized. All the relevant pre-
conditions were met. However, the KAP study results became
available at a later then expected stage of the Project
implementation.
2.2.3. Train patronage nurses
and Mahalla Advisors from
Women’s Committees to act
as facilitators in
The trainees should perceive participatory community activities as an
effective means to improve community health, sustain community
practices in households and getting themselves empowered.
Assumptions remained valid and fully realized. According to the
interviews, it appears that all pre-conditions are met.
78 IMCHS II Final Evaluation Report –August 31 / 2015
implementation of
community activities.
2.2.4. Create support groups
for promoting healthy
behaviors and monitoring of
outcomes of community
based BCC activities.
The community members should perceive the activity of support
groups as empowering rather than intrusive in the life of families and
households.
The support group members should be willing to volunteer rather than
seek remuneration from the project.
Assumptions remained valid and fully realized. According to the
interviews, it appears that all pre-conditions are met.
79 IMCHS II Final Evaluation Report –August 31 / 2015
Annex 10. Status of the achievement of the Project results and performance targets
Indicators Baseline Target Mid Term Review (Dec
2014)
Current status (June 2015) Prospect of Achievement
Outcome Indicators
1) # of MoH normative documents on MCH care sector reform adopted or updated according to international standards
0 10 3 approved 5 approved (3 submitted) Most Likely
2) % of targeted MCH care institutions that apply newborn survival and child survival packages recommended by WHO and UNICEF --- measured through the following sub-indicators:
2. a) % of targeted maternities that correctly apply newborn survival packages recommended by WHO/UNICEF
13.5%
>40% 13% Not known Less Likely
2. b) % of targeted paediatric hospitals/wards that correctly apply child survival packages recommended by WHO/UNICEF
2.3%
>25%
15%
Not Known More Likely
3) # of Medical Institutes which have integrated at least 70% of the newborn and child survival modules in the graduate and postgraduate curricula --- measured through the following:
3. a) # of medical institutes that updated their curricula according to the approved newborn and child survival subjects
0
8 8 8 Achieved
3. b) # of medical institutes with at least 70% of relevant teachers trained on approved newborn and child survival subjects
0
8
4
8
Achieved
4) Family and community interventions based on PLA approach on 5 key health behaviours institutionalized based on the results of the pilots
approach is not part of any policy
mechanisms institutionalized
PLA being piloted
PLA pilot is ongoing. Institutionalization Prospects are
unclear due to the low national level ownership
Output indicators
Result # 1
1) The MCHSSC, its secretariat, and JMT are established and fully functional
no official mechanism in place
At least 4 MCHSSC meetings and 4 JMT visits per year
3 MCHCC meetings and 4 JMT visits.
Total of 7 MCHCC meetings (out of planned 8) and 10 JMT visits Most likely
2) The quality insurance mechanisms are piloted and institutionalized, based on the project results
no mechanism in place
QI policy developed, tools and
The tool for certification health facilities
The tool for certification health facilities and methodology has been developed. The National
Less likely to be achieved before the Project end
80 IMCHS II Final Evaluation Report –August 31 / 2015
methodology piloted and submitted for MoH approval
and methodology has been developed. The National Concept on Quality Improvement (NCQI) drafted and discussed at the national consensus building workshop on 29-30 July 2014. At the moment the document is under finalization
Concept on Quality Improvement is finalized. The national team for the implementation is trained. While the target as defined is most likely to be achieved, it remains unclear whether the developed certification tool will be adopted and used.
3) Database of trainers and trainees on newborn and child survival packages developed and institutionalized
database in place for IMCHS1 trainers for project purposes only
database enriched, adjusted and institutionalized within Center of Medical Education under MoH
Database enriched, adjusted.
Database is available at the Centre of Medical Education, however not yet institutionalized
Most Likely
4) Best practices documented and shared with all interested parties
0 at least 2 publicly shared per year
Three (NMCR, BABIES,EMOC) best practices documented and shared during the cross-visit to Fergana region organized for regional level health managers and management staff of perinatal
One additional conference for dissemination of the best practices in hospital care for children was held in May 2015.
Achieved and exceeded
81 IMCHS II Final Evaluation Report –August 31 / 2015
centres from regions of IMCHS I and II.
Result #2
1. % of MCH care system managers and relevant health professionals who have implemented the gained knowledge on the newborn and child survival packages in their regular practice --- this indicator is measured through the following sub-indicators:
Will be obtained through the end-line assessment
1. a) % of MCH care system managers who have implemented the gained knowledge on the newborn and child survival packages in their regular practice
39% (in outpatient
care) 14 % (in inpatient
care)
60%
30 %
53%
41 %
Most Likely
1. b) % of children 2 month - 5 years that received medical care by GP doctors according to approved protocols/ standards (this sub-indicator is comprised of 3 dimensions as per below)
Will be obtained through the end-line assessment
i) % of children 2 m-5 years that were examined by GP doctors according to approved protocols/standards
23.1% 50% 57% Exceeded
ii) % of children under 2 years of age whose caretakers are asked by GP about breastfeeding, complementary foods, and feeding practices during this episode of illness
21% 50% 30% More likely
iii) % of children who do not need antibiotic and leave the facility without having received or having been prescribed antibiotics by GP
46.5% 70% 100% Exceeded
1. c) % of children 2 month-5 years whose caregivers received counselling by patronage nurses according to approved protocols/standards (this sub-indicator is comprised of 3 dimensions as per below)
Will be obtained through the end-line assessment
i) % of children (2 months - 5 years) with diarrhoea whose caretakers were counselled by patronage nurses (PN) to give extra fluid during illness
55.9% 70% 55% Not likely
ii) % of children 6-12 month age whose caretakers receive at list 3 recommendations on nutrition by PN
21% 50% 30% Less Likely
iii) % of children (2 months - 5 years), whose caretakers received at least three counselling messages from PN on when to return immediately
12.4% 30% 40% Exceeded
2. # of successfully functioning community BCC mechanisms which use the PLA approach
0 10 out of 12 No data Will be obtained through the end-
line assessment Most Likely
3. % of families with U5 years children, who have adopted five key healthy behaviours according to received training - this indicator is measured through these sub-indicators:
82 IMCHS II Final Evaluation Report –August 31 / 2015
3. a) % of caregivers able to recognize danger signs that require the child (0-2 months) to be seen by a doctor
31% 50%
No data Will be obtained through the end-
line assessment
Unclear
3. b) % of mothers who breastfed their child exclusively for the first 6 months of life
39.6% 50% Unclear
3. c) % of households providing a balanced diet for children, pregnant women and lactating mothers (this sub-indicator is comprised of 3 dimensions as per below)
i) % of children under 2 years of age reporting a minimum dietary diversity (% of children 6-23 months who consumed 4 or more food groups daily)
66.5% 75%
No data
Will be obtained through the end-line assessment
Unclear
ii) % of children fed with own produced vegetables 26% 35% Unclear iii) Average time parents and other caregivers in the household dedicate to the development of children under 5 years old
2 hours on average
3 hours on average
Unclear
iv) % of caregivers who have their children’s (under 5 years old) hands washed after going to the toilet.
