FINAL REPORT I Nigeria... · Management Group to give an in-depth insight that is well presented in...

FINAL REPORT I

© UNICEF

II FINAL REPORT

© UNICEF

FINAL REPORT This report was written by Universalia Management Group. The report represents the views of the consultant and should not be attributed to the UNICEF or to any other organization.

FINAL REPORT III

© UNICEF

Independent Evaluation of UNICEF Nigeria Training Investments Final Report © United Nations Children’s Fund, Nigeria, 2016. United Nations Children’s Fund Plot 617/618, UN House, Diplomatic Zone, Central Area District Abuja, Nigeria

UNICEF’s Nigeria Country Office manage evaluation and evaluative studies on a wide range of topics for the purpose of contributing to learning about what makes for effective development, as well as supporting accountability for results in Nigeria. These evaluations aim at identify what works and what does not in terms of achieving sustainable and equitable development results, and to throw light on how and why interventions succeed or not under various circumstances. In assessing UNICEF support to Nigeria government at Federal and State level and other development partners, these evaluations consider where, how and why progress is being made and the difference it is making in the lives of children, women in Nigeria. By publishing evaluation reports, the UNICEF Nigeria Country Office makes evaluation findings, lessons and conclusion available to a wide audience. Lessons learned from evaluation are expected to inform operational improvements and, ultimately, to support the achievement of better results. The publication of evaluation reports also supports accountability, by providing all interested parties with independently determined evidence relating to UNICEF’s performance. This provides a basis for informed dialogue and discussion, and helps to assure all stakeholders that the organization operates in an open and transparent manner. The content of this report do not necessarily reflect the policies or views of UNICEF. The text has not been edited to official publication standards and UNICEF accepts responsibility for error. The designations in this publication do not imply an opinion on the legal status of any country or territory, or of its authorities, or the delimitation of frontiers. The copyright for this report is held by the United Nations Children’s Fund – Nigeria Office. Permission is required to reprint/reproduce/photocopy or in any other way to cite or quote from this report in written form. UNICEF has a formal permission policy that requires a written request to be submitted. For non-commercial uses, the permission will normally be granted free of charge. Please write to the UNICEF Nigeria Country Office at the address below to initiate a permission request.

For further information, please contact: UNICEF Nigeria Country Office United Nations Children’s Fund Plot 617/618, UN House, Diplomatic Zone, Central Area District Abuja, Nigeria [email protected]

mailto:[email protected]

IV FINAL REPORT

© UNICEF

Foreword

UNICEF Nigeria is a learning organization, at the heart of the work we do, is ensure that, at the round of Country Program Document (CPD) 2018-2021 development, we address and take lessons as well as incorporate these into next programmme. Since training interventions are a major component of our work in Nigeria, the Country Office commissioned this evaluation to inform discussions on the next CPD. The findings show that, as an organization, we have invested considerably in Nigeria for the delivery of trainings to facilitate the implementation of UNICEF’s Country Programme in Nigeria. Trainings which are delivered by UNICEF in Nigeria have been appreciated by partners and participants for the relevance and quality of the trainings. However, the planning process is not part of an overarching country strategy, which leads to a diversity of trainings that runs the risk of dispersal. The absence of an explicit Theory of Change and the lack of a strong results measurement framework also do not permit adequate monitoring of training activities and their expected results. The recommendations are; 1) UNICEF in Nigeria should develop a comprehensive country training strategy to design trainings proactively, as opposed to focusing on individual punctual training activities that support specific aspects of programme implementation; 2) UNICEF in Nigeria should further monitor and evaluate its trainings by adopting monitoring and evaluation mechanisms and developing M&E instruments, guidelines and templates for training activities that systematically collect data on all training activities, expenditures and results and ; 3) UNICEF in Nigeria should ensure appropriate data management through the establishment and management of a database on trainings. We will address the recommendations and provide adequate responses. I wish to express my personal thanks to all sections within UNICEF Nigeria and especially to the Monitoring and Evaluation team for managing this evaluation and supporting Universalia Management Group to give an in-depth insight that is well presented in this report on the effectiveness of the UNICEF Nigeria training investment.

Mohamed Malick Fall Country Representative.

FINAL REPORT V

© UNICEF

Acronyms CD Capacity Development

CPD Country Programme Document

DAC Development Assistance Committee

DSA Daily Subsistence Allowance

FAO Food and Agriculture Organization

FGD Focus Group Discussion

FO Field Office

GM General Manager

HACT Harmonized Approach to Cash Transfers

IMEP Integrated Monitoring and Evaluation Plan

KM Knowledge Management

LGA Local Government Area

M&E Monitoring and Evaluation

NCO Nigeria Country Office

NGO Non-Governmental Organization

OECD Organization for Economic Cooperation and Development

RAM Results Assessment Modules

RBM Results Based Management

RWP Rolling Work Plan

ToC Theory of Change

TORs Terms of Reference

ToT Training of Trainers

UMG Universalia Management Group

UNDAF United Nations Development Assistance Framework

UNDAP United Nations Development Assistance Programme

UNDG United Nations Development Group

UNESCO United Nations Educational, Scientific and Cultural Organization

UNICEF United Nations Children’s Fund

WASH Water, Sanitation and Hygiene

WASHCOM Water, Sanitation and Hygiene Committee

WASHIMS WASH Information Management System

FINAL REPORT VII

© UNICEF

Contents

EXECUTIVE SUMMARY ................................................................................................. 1

1 INTRODUCTION ..................................................................................................... 5

1.1 Evaluation Objectives .................................................................................................................... 5

1.2 Evaluation Context ........................................................................................................................ 6

2 METHODOLOGY .................................................................................................... 8

2.1 Evaluation Phases .......................................................................................................................... 8

2.2 Approach to Data Collection ......................................................................................................... 8

2.3 Theoretical Approaches .............................................................................................................. 12

2.4 Methodological Limitations......................................................................................................... 12

3 EVALUATION FINDINGS ....................................................................................... 14

3.1 Evaluability Assessment .............................................................................................................. 14

3.2 Relevance .................................................................................................................................... 17

3.3 Effectiveness................................................................................................................................ 22

3.4 Efficiency ..................................................................................................................................... 39

3.5 Sustainability ............................................................................................................................... 40

3.6 Cross-Cutting issues .................................................................................................................... 41

4 CONCLUSION AND RECOMMENDATIONS ............................................................. 42

4.1 Conclusion ................................................................................................................................... 42

4.2 Recommendations ...................................................................................................................... 42

VIII FINAL REPORT

© UNICEF

Figures

Figure 2.1 Phases of the Evaluation .......................................................................................................... 8

Figure 2.2 Mapping of consulted UNICEF Offices ..................................................................................... 9

Figure 2.3 Number of respondents disaggregated by sector and region ............................................... 11

Figure 2.4 Respondents’ participation in a UNICEF training between 2014 and May 2016 ................... 11

Figure 2.5 Respondents’ participation in a UNICEF training before 2014 .............................................. 11

Figure 2.6 Gender of respondents .......................................................................................................... 11

Figure 2.7 Occupation of respondents at the time of the training ......................................................... 11

Figure 3.1 Participants’ needs assessment prior to the training ............................................................ 19

Figure 3.2 Respondents’ needs assessment ........................................................................................... 20

Figure 3.3 Training response to respondents’ needs .............................................................................. 20

Figure 3.4 Relevance to participants’ work/ main occupation ............................................................... 21

Figure 3.5 Training’s response to respondents needs to do their job better ......................................... 21

Figure 3.6 Respondents’ selection to attend the UNICEF training ......................................................... 21

Figure 3.7 Clarity of Respondents about the purpose of the training prior to their attendance ........... 21

Figure 3.8 Reasons why respondents attended the UNICEF training ..................................................... 22

Figure 3.9 Quality of presentation and lecturing .................................................................................... 26

Figure 3.10 Usefulness of training material during the training sessions ................................................. 27

Figure 3.11 Usefulness of training material after the training sessions ................................................... 27

Figure 3.12 Extent to which objectives of the training were clearly outlined at the start of the session 31

Figure 3.13 Respondents level of engagement was during the training .................................................. 31

Figure 3.14 Participants’ perception of whether the training number of participants was optimal to ensure everyone’s participation ............................................................................................ 32

Figure 3.15 Participants’ perception of whether the training was worth the time invested ................... 32

Figure 3.16 Participants’ perception of whether the training was satisfactory ....................................... 32

Figure 3.17 Duration of trainings attended .............................................................................................. 32

Figure 3.18 Participants’ perception that they were given adequate opportunity to practice learning .. 33

Figure 3.19 The extent to which content of the training met respondents needs ................................... 33

Figure 3.20 Respondents perceptions whether they acquired the knowledge, skills and capacities they had expected .......................................................................................................................... 34

Figure 3.21 Participants’ overall feeling that respondents’ capacities have increased ............................ 34

Figure 3.22 Feeling of commitment by respondents to apply what they learned in their work / life ..... 34

Figure 3.23 Use of newly acquired knowledge/skill by respondent ......................................................... 36

Figure 3.24 UNICEF training changes to respondents’ ways of doing things ........................................... 36

Figure 3.25 Respondents’ perception of adequacy of support from their organizations to apply newly acquired knowledge ............................................................................................................... 36

Figure 3.26 Respondents organizations’ openness to changes suggested as a result of participation in the training ............................................................................................................................. 36

FINAL REPORT IX

© UNICEF

Figure 3.27 Use of training learning to the benefit of participants’ communities ................................... 38

Figure 3.28 Use of learning to the use of participants’ work .................................................................... 38

Figure 3.29 Respondents formally train other people on the acquired skills / knowledge ...................... 41

Figure 3.30 Respondents share informally what the acquired skills / knowledge with other people ..... 41

Figure iv.1 Five Key UNDG Steps for Capacity Development .................................................................. 52

X FINAL REPORT

© UNICEF

Tables

Table 2.1 Key stakeholders consulted and approach to data collection ............................................... 10

Table 2.2 Methodological challenges and mitigation strategies ........................................................... 12

Table 3.1 Compliance of the actual ToT model with international practices ........................................ 24

Table 3.2 Availability of Information: Rick Davies Checklist .................................................................. 29

Table 3.3 Availability of M&E data on trainings at the field level, aggregated by sector...................... 30

Table 3.4 Changes in training participants’ behaviour captured in the survey ..................................... 37

Table 4.1 Table of recommendations .................................................................................................... 43

Table v.1 Interviews conducted during the inception phase ................................................................ 54

Table v.2 Interviews conducted during data collection: Training Planners/Implementers interviewed ............................................................................................................................ 54

Table vi.1 Basic Information on the Training Activity ............................................................................. 57

Table vi.2 Persons to Consult ................................................................................................................. 58

Table vi.3 Documents to Consult ............................................................................................................ 58

Table vi.4 Key Questions by Phases of a Training Activity ...................................................................... 59

Appendices

Appendix I List of Findings ......................................................................................................... 44

Appendix II Evaluation Matrix ................................................................................................... 45

Appendix III Roles and Responsibilities of Team Members ........................................................ 50

Appendix IV Theoretical Frameworks ........................................................................................ 52

Appendix V List of Persons Consulted ........................................................................................ 54

Appendix VI Training Evaluation Template ................................................................................ 57

Appendix VII Data Collection Tools ............................................................................................ 60

Appendix VIII Bibliography ........................................................................................................ 77

Appendix IX List of Trainings ...................................................................................................... 92

Appendix X Table of Changes to the Original Evaluation Questions in the TORs ........................ 97

Appendix XI The Training Cycle: From Planning to Result ........................................................ 100

Appendix XII Terms of Reference ............................................................................................ 101

FINAL REPORT 1

© UNICEF

Executive Summary This evaluation of UNICEF Nigeria’s training investments was commissioned by the UNICEF Nigeria’s country monitoring and evaluation unit to determine the extent to which individual training activities conducted by UNICEF in Nigeria have been relevant, effective, efficient and sustainable in supporting the implementation of the Country Programme Documents (CPD) 2014-2017.

This evaluation covers Investments by UNICEF Nigeria into ‘Individual Training Activities’ between January 2014 and May 2016 in the five CPD ‘sub-components’ (also called ‘sections’ or ‘sectors’) of Health, Nutrition, Education, Child Protection, and WASH.

The objectives of the evaluation are:

To strengthen UNICEF Nigeria’s capacity development strategy with clear recommendation on how to improve future trainings and its implementation;

To provide information on the quality of trainings provided by UNICEF: content, delivery, timeliness, felt need, etc.; and

To contribute to knowledge management and organizational learning, by providing the evidence to support best practices on trainings.

The evaluation adopted mixed methods for the data collection and was conducted in six phases:

Launch call (August 2016),

Evaluability Assessment (September 2016)

Inception mission to UNICEF Nigeria’s offices in Abuja and visits to six field offices to establish data sources (September 2016)

Data collection in eight States in Nigeria (November 2016-January 2017)

Data analysis and report writing (February-April 2017)

Validation of findings and submission of final report (April 2017)

Evaluation findings were mixed in terms of relevance. Beyond the various notable efforts to meet the national training needs, UNICEF Nigeria’s Training Investments were not designed as a coherent programme and lack an explicit theory of change. In the absence of a national capacity gap analysis, trainings are scattered, which undermines relevance of the portfolio of training investments as a whole.

Also on relevance, UNICEF Nigeria’s individual training activities respond to a certain extent to the needs of participants with opportunities for some possible improvements. In that sense, surveyed training beneficiaries were overall satisfied with the training response to their needs and to their jobs. However, the current training needs identification and participant selection processes give no insurance as to how targeted its portfolio of trainings is in terms of selection of beneficiaries.

In terms of effectiveness, the evaluation the evaluation noted the absence of a comprehensive national training strategy where trainings are strategically planned to respond to national training needs. Also, the evaluation observed the absence of a robust results framework for its trainings, in order to assess whether trainings have been delivered properly, and to whom. In the absence of a country training strategy and results framework, the overall effects of the training portfolio’s impact are jeopardized and run the risk of dilution of potential results.

2 FINAL REPORT

© UNICEF

From a design perspective, all sectors except the Child Protection sector adopt the same approach of delivering trainings through Training of Trainers (ToT). As ToT activities, the effectiveness of the training activities is mixed. Training participants revealed that respondents were satisfied with the quality of the technical trainings received. However, in the actual model, trained trainers are expected to return to their respective State to develop and deliver training courses to support implementation. As it stands, UNICEF Nigeria’s individual training activities limit themselves to delivering a solid technical component. The trainings do not include any modules on training process, or how to deliver trainings, nor does it provide training participants with the opportunity to practice their teaching skills prior to delivering trainings. Indeed, at the moment, the training is based on the assumption that participants will be able to deliver training after attending a technical workshop. This assumption could be true in some cases, but needs to be verified.

In addition, the current training design does not have a quality assurance mechanism, which represents a risk to training effectiveness. It does not proceed to any qualification test or ‘certification’ process of its trainees to identify the extent to which they can deliver trainings effectively, nor does it provide additional learning opportunities on the subject matter for the trainees to update their knowledge. In that sense, cascaded trainings run the risk of quality degradation.

In terms of monitoring and evaluation, some efforts are made to monitor training participants’ learning at the field level. However, the monitoring of participants’ reaction, learning, changes in behaviour and impact is not systematic. One sector was found to be pro-active in monitoring of trainings: WASH was noted for the availability of monitoring data during the Evaluability Assessment and individual interviews.

Based on all data collected and analyzed training activities, there is a high potential for effectiveness in terms of reaction and changes in behaviour. Survey respondents found the training to be favorable and engaging where they acquired useful knowledge. The evaluation also collected positive feedback of the stories of changes in participants’ behaviour, mainly in terms of personal confidence, technical work, and work with community, planning, M&E and reporting, management and training/class delivery.

In terms of efficiency, the evaluation concluded that the ToT approach is a contributing factor to efficiency. However, the very limited and incomplete data on the UNICEF training activities seriously undermines UNICEF’s ability to ensure that trainings are performing best. According to self-reported evidence, the potential for the sustainability of training results is strong. There is however some room for improvement, especially in terms of quality assurance to ensure that cascading trainings remain of quality. Trainings sustainability was also observed to be dependent on Programme delivery. If a Programme stops, its training components end.

In terms of gender, on the dimensions examined in this evaluation, there is not enough evidence or indications as to whether the gender component was taken into account in the training design, nor whether specific actions were taken in order to ensure equal opportunities for women and men to participate in the trainings, in terms of access, participation and community engagement. This evaluation did not analyze the content of the training to assess whether the gender component was taken into consideration.

In conclusion, a considerable effort is put forth by UNICEF in Nigeria for the delivery of trainings to facilitate the implementation of UNICEF’S Country Programme in Nigeria. Trainings which are delivered by UNICEF in Nigeria are appreciated by partners and participants for the relevance and quality of the trainings. However, the planning process is not part of an overarching country strategy, which leads to a diversity of trainings that runs the risk of dispersal. The absence of an explicit Theory of Change and the lack of a strong results measurement framework also do not permit adequate monitoring and learning.

FINAL REPORT 3

© UNICEF

While some efforts are made to monitor training participants’ learning at the field level, the monitoring and reporting on trainings is not systematic, accompanied by the absence of a centralized knowledge management system.

From a training design perspective, the training of trainers can be a good model if appropriately designed. Survey of past participants revealed positive effects in terms of reaction, learning, changes in behaviour and knowledge transfer. However, the evaluation observed that the current design might affect effectiveness, being based on a false assumption that delivering technical trainings without training on training process would allow participants to become effective trainers. Follow-up and quality assurance mechanisms are also subject to improvement, to reduce risks to efficiency and sustainability.

The evaluation team developed a total of three prioritized and sequenced recommendations (outlined below) that are based on its findings and conclusions. These recommendations are the following:

Recommendation 1: UNICEF in Nigeria should develop a comprehensive country training strategy to design trainings proactively, as opposed to focusing on individual punctual training activities that support specific aspects of programme implementation.

Recommendation 2: UNICEF in Nigeria should further monitor and evaluate its trainings by adopting monitoring and evaluation mechanisms and developing M&E instruments, guidelines and templates for training activities that systematically collect data on all training activities, expenditures and results.

Recommendation 3: UNICEF in Nigeria should ensure appropriate data management through the establishment and management of a database on trainings

FINAL REPORT 5

© UNICEF

1 Introduction 1.1 Evaluation Objectives

1. The purpose of this evaluation is to determine the extent to which individual training activities conducted by UNICEF in Nigeria have been relevant, effective, efficient and sustainable in supporting the implementation of the Country Programme Documents (CPD) 2014-2017. To the extent that the expected findings contribute towards enhancing UNICEF’s readiness to reliably measure, monitor, evaluate and report on progress, these also contribute towards UNICEF’s organizational and development effectiveness. This is a formative evaluation whose objective is to inform future work carried out by UNICEF in Nigeria. Primary users of this evaluation include UNICEF Nigeria’s Deputy Representative, Field Offices, programme managers and UNICEF Nigeria country monitoring and evaluation unit.

2. Specifically, and as cited in the TORs, the objectives of the evaluation are:

To strengthen UNICEF Nigeria’s capacity development strategy with clear recommendation on how to improve future trainings and its implementation;

To provide information on the quality of trainings provided by UNICEF: content, delivery, timeliness, felt need, etc.; and

To contribute to knowledge management and organizational learning, by providing the evidence to support best practices on trainings.

3. This evaluation covers Investments by UNICEF Nigeria into ‘Individual Training Activities’ between January 2014 and May 2016 in the five CPD ‘sub-components’ (also called ‘sections’ or ‘sectors’) of Health, Nutrition, Education, Child Protection, and WASH.

4. This is an evaluation led by and for the UNICEF Nigeria Country Office (NCO) as a whole. Although the evaluation recognizes important differences between CPD sections and took these into account throughout the assignment, the scope of this evaluation does not include a full section-by-section review of training activities, but rather an overall assessment of UNICEF NCO training investments.

Terminology

5. Below, are defined key terms as understood in the context of this evaluation:

By Capacity, it means “the numbers of people (staff) and resources within an organization for the performance of specific functions, as well as the capability and ability of the staff and resources to perform those functions.” (cf. TORs, p.3). It is worth noting that the Capacity to carry out a task does not necessarily imply the willingness, practice, habit or behaviour of actually doing it.

By Capacity Development (CD), it means increasing the ability of individuals, communities or organizations to execute certain tasks. CD can either communicate the same skills to a wider set of actors, or instruct a given set of actors on how to better deploy and employ their existing skills; for instance, by modifying the institutional, societal and environmental context.

By Individual Capacity Development, it means ‘training,’ that is, to develop, through specialized instruction and practice, the skills and abilities of individuals to carry out certain tasks.

6 FINAL REPORT

© UNICEF

‘Training,’ ‘Training Session,’ ‘Training Activity,’ and ‘Individual Capacity Development’ are henceforth used interchangeably.

The Country Programme Document’s (CPD) sub-components (e.g. ‘Nutrition’) are referred to as ‘sectors,’ and the UNICEF teams responsible for managing these are referred to as ‘sections.’

6. The present report presents evaluation findings that respond to questions presented in the TORs, as presented in the Evaluation Matrix (see Appendix II).

Structure of the Report

7. The report is organized into four sections:

Section 1 – Introduction, including evaluation objectives and evaluation context

Section 2 – Methodology, including evaluation phases, approach to data collection, theoretical approaches and methodological limitations

Section 3 – Evaluation Findings on Evaluability, Relevance, Effectiveness, Efficiency, Impact and Sustainability of the Project

Section 4 – Conclusions and Recommendations.

8. The appendices contain additional documents, including the terms of reference, the evaluation matrix, the list of stakeholders consulted, the list of documents reviewed, the evaluation frameworks and the data collection tools.

1.2 Evaluation Context

9. Based on conversations and documents reviewed, it is understood that the rationale for this evaluation is grounded in the following considerations:

From the perspective of the UNICEF Nigeria Country Office (NCO), training investments are substantial and cut across most (CPD) sectors, but have never been evaluated as such. The Monitoring and Evaluation (M&E) unit of the NCO would like to better understand the relevance, effectiveness, efficiency, sustainability, and impact of these training investments, for both accountability and learning purposes, and especially in view of the development of the next CPD.

From the perspective of middle-income countries, Capacity Development is becoming the primary and preferred contribution of international partners, including UNICEF (cf. the UNICEF Programme, Policy and Procedure Manual on “Programming in Middle-Income Countries”, pp.61-62). Middle-income countries like the Federal Republic of Nigeria aspire to autonomously provide for the rights and needs of their citizens, and are working with partners to identify and address capacity gaps.1

From a global perspective, Capacity Development is increasingly becoming a hallmark of development programming, and is aligned with the Paris Declaration on Aid Effectiveness, as well as with the 2030 Agenda for Sustainable Development. At the level of UNICEF, Capacity Development is one of the seven “implementation strategies” of the UNICEF global Strategic Plan

1 As of July 2016, the World Bank classifies Nigeria as a “Lower middle income” country. See siteresources.worldbank.org/DATASTATISTICS/Resources/CLASS.XLS, accessed September 21st, 2016.

file:///C:/Users/bisahismail/AppData/Local/Microsoft/Windows/Temporary%20Internet%20Files/Content.Outlook/O227DFN3/siteresources.worldbank.org/DATASTATISTICS/Resources/CLASS.XLS

FINAL REPORT 7

© UNICEF

2014-2017 (p.11); at the level of the United Nations Development Group, CD is one of the five “principles” of Country Programming.2 Thus, understanding whether CD works and how it could be improved is of high importance for future UNICEF programming in Nigeria, and elsewhere.

2 Cf. UNDG Guidance Note: Application of the Programming Principles to the UNDAF.

8 FINAL REPORT

© UNICEF

2 Methodology 2.1 Evaluation Phases

10. The evaluation included five main phases, as presented in Figure 2.1 below.

Figure 2.1 Phases of the Evaluation

11. Following (a) two launch conference calls with UNICEF Nigeria’s M&E division (August 2016); (b) an assessment of information available and missing, presented in the Evaluability Assessment (September 2016); (c) a three-day inception mission to UNICEF Nigeria’s offices in Abuja, including phone conversations with field offices (September 2016); (d) visits to six field offices to establish data sources and data collection in eight States in Nigeria3 (November 2016-January 2017); (e) data analysis was conducted and followed by report writing (February-April 2017). (f) A debrief presentation was conducted in UNICEF Nigeria country office to validate the evaluation’s findings and conclusions and collect stakeholders’ perceptions and feedback. The report was then revised to its final version (April 2017).

12. This evaluation is aligned with the obligations of evaluators relating to independence, impartiality, credibility, conflicts of interest, and accountability.

13. The following section outlines the approach adopted to complete this assignment.

2.2 Approach to Data Collection

14. The data-collection phase constitutes the core of the work for this evaluation. In order to gather information on training investments, four data-collection strategies were adopted: document review, interviews, focus group discussions, and a questionnaire survey.

15. The first – document review – involved desk-based work, whereas the three remaining strategies involved site visits.

Sites Sampling

16. Key “sites” for this evaluation are 10 of UNICEF Nigeria’s Field Offices,4 from which the majority of training investments are managed. Sampling of the Field Offices included seven States.

17. At the inception and evaluability assessment phases, sampling was limited to six Field Offices hoping that training activities in the five sectors under review will be covered. However, during the first field visit to establish the data sources, the evaluation team discovered that there were States with high training concentration. States sampling was therefore modified to include the following 10 States:

3 (Appendix V lists the persons consulted) 4 Field Offices are sometimes called ‘Zonal Offices’ in UNICEF Nigeria documentation. We use these interchangeably.

InceptionEvaluabilityAssessment

Data Collection

Data Analysis

Validation of Findings

Reporting

FINAL REPORT 9

© UNICEF

Abuja, where the NCO is located, where the team conducted high-level interviews and focus group discussions (FGDs) with senior management.

Enugu, Lagos5, Bauchi, Kaduna, Sokoto, Katsina, and Akure, where the team conducted interviews and FGDs with staff, training participants and beneficiaries, and administered survey questionnaires to training participants.

Rivers and Borno were not included in the data collection 6for two reasons: in the case of Rivers because of its proximity to Enugu FO, and in the case of Borno due to the ongoing security situation in the area.

Niger and Anambra, which are not Field Offices, where the team conducted interviews and FGDs with training participants and beneficiaries, and administered survey questionnaires to training participants.

18. The following map (Figure 2.2) presents the sample of the visited States.

Figure 2.2 Mapping of consulted UNICEF Offices

Desk Review

19. The desk review served two purposes. First, a review of general, strategic, and guidance documentation helped further understand and refine the context of UNICEF’s training investments, and in particular (a) their relevance in the face of capacity gaps in Nigeria; (b) their alignment with UNICEF Headquarters and United Nations System Capacity Development Guidance; and (c) their alignment with NCO Country Programme Strategies. This included documents such as UNDG Capacity Development Guidelines, UNICEF’s Global Strategic Plan, UNICEF’s Programme Policy and Procedure Manual, the Nigeria UNDAF and UNDAP, the NCO Country Programme Document, NCO Rolling Work Plans and Rolling Management Plans, IMEP data, RAM data, and so forth. The full bibliography consulted is presented in Appendix VII. Second, the desk review provided key information with regards to the evaluation of individual training activities.

5 The Lagos FO has recently been turned into an NCO Zonal Office, and all of its former FO responsibilities have been transferred to the new Akure FO. Interviews/FGD for Lagos were conducted at the Akure FO except for child protection, where the interview, FGD and questionnaire survey were done in Lagos. 6 The two remaining ‘new’ Field Offices (Rivers and Borno) were not included for two reasons: in the case of Rivers because of its proximity to Enugu FO, and in the case of Borno due to the ongoing security situation in the area.

10 FINAL REPORT

© UNICEF

Individual interviews

20. Individual interviews were used for stakeholders who were likely to have substantive strategic or operational insights on training investments (in general, or related to a specific training activity), such as UNICEF Programme Specialists, Chiefs of Field Offices, Sector Chiefs, and Government Officials. Interviews were conducted in a semi-structured fashion, on the basis of questions which aimed to contribute information to key evaluation questions. A sample interview protocol is presented in Appendix VII.

Survey Questionnaire & Focus Group Discussions

21. Survey questionnaires and focus group discussions primarily targeted training participants and training beneficiaries.

a. The first category includes persons who actually attended training sessions organized, facilitated or financed by UNICEF Nigeria; these include, depending on the kind of trainings, various government officials, civil servants, service providers (such as teachers and nurses), third parties (e.g. journalists, NGO representatives), and community members, workers, volunteers and leaders.

b. The second category includes stakeholders who are expected to benefit from the conduct of the training: beyond participants themselves, this includes managers of capacitated institutions, recipients of capacitated services, and other relevant affected organizations and partners.

22. In order to administer questionnaires and conduct focus group discussions, training participants and beneficiaries were sampled prior to being reached and successfully consulted. In order to address the absence of a definitive list of training (including by FO), the absence of complete training participant and beneficiary lists, and the unsure ability to contact respondents, a first round of field visits were conducted to establish data sources for training participants from which respondents were randomly selected for sampling.

23. Table 2.1 below summarizes the general outlay of data collection.

Table 2.1 Key stakeholders consulted and approach to data collection CATEGORY OF

STAKEHOLDERS KEY PERSONS MAIN APPROACH

Planners of Training

UNICEF staff: section chief, section specialists (including at FO level), M&E specialists, financial specialists Government officials at national / state / LGA level

Interviews

Designers of Training

UNICEF section specialists, M&E specialists, financial specialists Relevant experts (UNICEF, partners, consultants, contractors)

Interviews

Deliverers of Training

UNICEF section specialists Relevant experts (UNICEF, partners, consultants, contractors)

Interviews

Recipients of Training

Training participants from government, partners, communities (both ToT and Step-down, if applicable)

Survey Questionnaires (Paper and Focus Group Discussions)

Beneficiaries of Training Programme

Training participants Managers of capacitated institutions Affected citizens, communities Affected organizations, partners

Survey Questionnaires (Paper) Focus Group Discussions

24. Five focus groups discussions were held with average of 5-7 participants, 34 respondents were interviewed (26 UNICEF Staff and 8 government officials). 506 respondents completed the survey. The

FINAL REPORT 11

© UNICEF

following Figures (Figure 2.3 to Figure 2.7) outlay the Descriptive Statistics of the survey respondents’ sample. Figure 2.3 Number of respondents disaggregated by sector and region

Figure 2.4 Respondents’ participation in a UNICEF training between 2014 and May 2016

Figure 2.5 Respondents’ participation in a UNICEF training before 2014

Figure 2.6 Gender of respondents Figure 2.7 Occupation of respondents at the time of the training

42

2

40

0

28

149

75

88

82

0

Health

Nutrition

WASH

Education

Child Protection

South North

11%

15%

27%

6%

41%

2014

2015

2016

Do notremember

Between 2014to 2016

25%

42%

25%

8%Yes, one

Yes, Several

No, Never

Not sure/ Donot remember

67%

33%

Male

Female

86%

3%7%

1%

3%Employed inpublic sector

Employed inprivate sector

Self-employed

Unemployed

In school/studying

12 FINAL REPORT

© UNICEF

2.3 Theoretical Approaches

25. The adopted approach to assessing UNICEF Nigeria’s training investments is underpinned by the following approaches and standards: OECD-DAC criteria7 to assess the relevance, effectiveness, efficiency and sustainability of the NCO’s

training investments; United Nations Development Group’s Guidance (UNDG) on Capacity Development (CD)8, defining

Partner Engagement, Needs Assessment, Design of CD Strategy, Implementation of CD Activities, and Evaluation of CD Activities as the five main steps to CD;

Kirkpatrick Model9 for evaluating the effectiveness of training, building on four consecutive levels of training ‘results’: reaction, knowledge and skills, change in behaviour and results; and

Cross-Cutting Issues: In addition to these three frameworks, the evaluation also followed four cross-cutting considerations in its approach: differences by programme section, differences by type of training, and gender and equity were taken into consideration.

26. For further information on each of the frameworks, please refer to Appendix IV. These frameworks are all integrated into the Evaluation Matrix (see Appendix II), and reflected in data-collection tools (see Appendix VII). 27.

2.4 Methodological Limitations 28. The following key observations, highlighted in Table 2.2. are helpful to understand how the evaluation team approached this evaluation.

Table 2.2 Methodological challenges and mitigation strategies

OBSERVATIONS IMPLICATION FOR THE EVALUATION METHODOLOGY

There are no overarching objectives for training investments (or, these cannot be differentiated from the overarching objectives of the CPD as a whole).

Therefore, many evaluation findings relied on an aggregation of findings related to individual training activities. UNICEF’s training investments are evaluated as a set of activities, rather than as one coherent programme.

Lack of data to triangulate findings on effectiveness and to assess efficiency Available documentation does not provide any indication of the training targets (e.g., expected number of trainings to be delivered, number of training participants, duration of trainings, participants, etc.). Also, information on the nature, type, and results of delivered training activities is not collected systematically.

In the absence of planned explicit targets and monitoring data, effectiveness of trainings cannot be evaluated, including the extent to which planned training activities were delivered or the extent to which the expected outputs were reached (number of trainings, number of training participants, etc.). Also, survey results regarding effectiveness in terms of learning, changes in behaviour and results cannot be triangulated.

7 The DAC Principles for the Evaluation of Development Assistance, OECD (1991), Glossary of Terms Used in Evaluation, in ‘Methods and Procedures in Aid Evaluation’, OECD (1986), and the Glossary of Evaluation and Results Based Management (RBM) Terms, OECD (2000).

8 These Capacity Components are described in the UNDG Capacity Assessment User Guide, Feb 2008: “The eight capacity components are: human resources; public sector accountability; access to information, development knowledge and technology; inclusion, participation, equity and empowerment; financial resources; material resources; environmental resources; and external/international relations” (p.5) 9 For a basic introduction to the Kirkpatrick model, and for the source of the Kirkpatrick sub-questions in this table, see http://www.kirkpatrickpartners.com/OurPhilosophy/TheKirkpatrickModel (accessed September 22nd, 2016).

http://www.kirkpatrickpartners.com/OurPhilosophy/TheKirkpatrickModel

FINAL REPORT 13

© UNICEF

OBSERVATIONS IMPLICATION FOR THE EVALUATION METHODOLOGY

In order to mitigate this challenge, the evaluation took note of self-reported data and did not take its reliability for granted. Whenever possible, the evaluation team used its own judgement and information collected through interview and focus group discussions, to filter the information received.

Lack of data to assess efficiency Financial data on trainings is unavailable.

Cost efficiency could not be assessed due to the lack of financial data.

14 FINAL REPORT

© UNICEF

3 Evaluation Findings

3.1 Evaluability Assessment

Finding 1: UNICEF Nigeria’s Training Investments were not designed as a coherent Programme and lack a theory of change. These shortcomings are problematic and allow limited opportunities for making an informed judgment on the portfolio overall

Training investments were not planned as a Programme

29. The first step in determining the evaluability of an intervention or set of investments is to clearly delineate its scope. During the evaluation exercise, it was observed that Capacity Development (and its ‘individual-level’ counterpart, training) has not been planned as an integrated or cross-cutting project or Programme. Instead, every Section requests training as it relates to the implementation of a particular intervention where training is almost exclusively conducted at the activity level. This impacted the way the evaluation was conducted, where many evaluation findings relied on an aggregation of findings regarding individual training activities. This has implications on the feasibility for UNICEF’s training investments to be evaluated as one coherent Programme, rather than a set of activities.

Absence of a comprehensive database of training activities

30. It was also observed that a comprehensive database for training activities conducted does not exist. UNICEF was not able to provide a specific or an estimate of the number of training activities that have been conducted in the period under review (2014–May 2016).

31. During the evaluability assessment and data collection phases, the evaluation team attempted to reconstruct a list of training activities conducted in Nigeria, based on document review (Work Plans and Annual Reports), interviews and survey results. (Please refer to the full list of trainings captured in Appendix IX).

32. The evaluation included in this universe of training all individual capacity development activities (i.e., ‘training’) listed in the 2014-2015 or 2015-2016 Rolling Work Plans of the five sectors named in the TORs: Health, Education, Nutrition, Child Protection, and WASH.

33. Although Rolling Work Plans (RWP) were used as a basis for mapping out the universe of training for this evaluation, they are not ideal sources of information for the following reasons: (a) the format and information provided in RWPs varies from sector-to-sector, and from year-to-year even within a sector; (b) RWPs often describe ‘training’ indirectly, instead of referring to general ‘capacity development’ or ‘technical assistance’; (c) RWPs often group together various activities under one heading. Consequently, it was not always possible to identify with precision what kinds of training were planned, in terms of objectives, types of participants, numbers, region, and budget.

34. To mitigate information gaps, the evaluation cross-referenced the list of training drawn from RWPs with three further sources: (a) survey results; (b) training reported in the Annual NCO Reports for 2014 and 2015. However, the narrative style of annual reports is not systematic in the information it reports about training.; and (c) other training documents shared by the Evaluation Unit for the purpose of this evaluation.

FINAL REPORT 15

© UNICEF

35. This evaluation, in particular:

Includes training of various types, sizes, geographical coverage, partners and objectives.

Includes training conducted between January 2014 and May 31st, 2016 (as specified by the TORs); excludes training conducted before or after these dates.

Includes training on emergency operations where these training are classified by the Country Programme Document as activities under the five aforementioned sectors; but excludes training on emergency operations conducted under Outcome 13 (Emergency) of the CPD.

Excludes UNICEF internal training (such as UNICEF Staff training).

Excludes outreach, C4D, advocacy and awareness-raising activities which do not involve a sustained teaching component, with well-defined materials, trainers, and participants. For instance, ‘Global Handwashing Day’ and similar activities are not considered “training” for the purposes of this evaluation; although these activities share certain aspects with “training” (educational message, targeted behaviour change), they do not offer the active support and lessons usually provided in training.

Absence of a Theory of Change

36. The evaluation was able to confirm that there was no overarching causal chain that connects UNICEF training investments with intended results, other than their incorporation as ‘activities’ in the CPD, and corresponding sectional RWPs.

37. This evaluation attempted to draft a Theory of Change (ToC) that would reflect the current understanding of training interventions based on documents and interviews carried out.

38. The proposed ToC is based on Kirkpatrick’s model to evaluate training. Its four levels are shown on the left-hand side, e.g., Level 1 – Reaction, Level 2 – Learning, etc. For each level, a result was formulated that should be expected from UNICEF training investments. Results for the first three levels were drawn from the evaluation understanding of the Kirkpatrick model and UNICEF’s interventions, while the results for the fourth level are based on the UNICEF’s capacity development strategy.10 The final impact or goal to be achieved was extracted from the UNICEF Nigeria Country Programme Document.

39. Whenever relevant, the evaluation identified assumptions linked to results and also included examples of external influences which may apply to any of the four levels described above.

40. This draft ToC should be discussed internally, refined and finalized by the UNICEF Nigeria CO. The evaluation made a conscious decision not to include additional, embedded ToCs in order to avoid too much complexity (e.g., an embedded ToC for the training of trainers).

10 “Capacity development aims to (a) strengthen delivery of supplies and services; (b) develop the enabling environment needed to guarantee key rights, promote behaviour change and demand for services; and (c) strengthen the ability of rights-holders to claim key rights.” (p.11, UNICEF Strategic Plan 2014-2017).

16 FINAL REPORT

© UNICEF

Exhibit 3.1 Draft Theory of Change

FINAL REPORT 17

© UNICEF

3.2 Relevance

41. The evaluation adopted the OECD DAC’s Glossary of Key Terms in Evaluation and Results-Based Management to define Relevance as being “The extent to which the objectives of a development intervention are consistent with beneficiaries’ requirements, country needs, global priorities and partners’ and donors’ policies.”11

42. Overall, UNICEF Nigeria’s individual training activities are seen as relevant by training attendees. However, their relevance of the overall portfolio, in the absence of country training strategy and country capacity gaps assessments, is questionable.

Finding 2: Beyond the various notable efforts to meet the national training needs, in the absence of a national capacity gap analysis, trainings are scattered, which undermines relevance of the portfolio of training investments.

43. There have been various notable efforts to meet the national training needs, with the objective of facilitating the implementation of the UNICEF Nigeria Country Programme.

44. Capacity Development (and training) is a priority in UNICEF’s (2014-2017) strategic plan where “Capacity development at individual, community and government levels has always been one of the main UNICEF implementation strategies. Support to individual and community capacity development is often provided through ‘communication for development’ to strengthen the capacity of government at all levels, UNICEF focuses on increasing capacity to address gaps of rights-holders and duty-bearers. Measures include training and technical assistance for reforms of government institutions; piloting models for scaling up; and using national and local systems and processes to strengthen capacities and empower children,

11 Glossary of Evaluation and Results Based Management (RBM) Terms, OECD (2000) Page 32.

18 FINAL REPORT

© UNICEF

families and communities. Strengthening management systems, results-based management and planning capacities are key components of a comprehensive approach to capacity development.”12

45. UNICEF Nigeria’s trainings are also aligned with UNICEF’s priorities, where capacity development is mentioned in the Country Programme (2014-2017) as being one of the four interlinked strategies to reduce disparities and reach the most disadvantaged children: “Developing technical capacities at State and LGA levels to coordinate the delivery of quality social services”13. In addition, all UNICEF training activities in Nigeria are conducted as part of programme delivery which makes them fit for purpose. In terms of alignment with national strategic orientations, Nigeria produced in December 2009 a national development strategy entitled Vision 20:2020, where “investing in human capacity development” is a key target to support the Vision 20:2020 aspirations. UNICEF’s Country Programme is aligned with Vision 20:2020 national priorities: “The (UNICEF) Country Programme was developed through close collaboration with the Federal Government of Nigeria, led by the National Planning Commission. Its preparation was synchronized with the United Nations Development Assistance Framework (UNDAF-III) 2014-2017, for which the United Nations country team chose the One Programme modality. The preparation was guided by Vision 20:2020.14”

46. In order to identify training needs, UNICEF engages in a consultative process with its implementing partners. UNICEF consults national institutions and agencies to plan, design and deliver its training programmes. Prior to training planning and delivery, each sector systematically agrees with the State implementing partners on specific training that would be required to deliver a particular programme.

47. The process through which training is programmed can be described as follows15: If a particular intervention requires training to be delivered, then a draft proposal is submitted by an implementing partner to UNICEF for approval. If the proposal is approved by UNICEF, training planning is conducted by the latter through a wide consultation with the relevant stakeholders (at the national, state and ward levels) in order to define the training objectives and targeted participants.

48. In addition to the process of engaging partners described above, specific training needs are then identified using one of the following approaches:

Consultation with relevant stakeholders: Meetings/workshops with national stakeholders are held to agree on what training is needed in each sector.

Observation of errors made by the health workers in the execution of their duties.

Pilot surveys: some trainings are informed by findings and recommendations of pilot surveys conducted by UNICEF to collect internal and/or partner’s staff training needs. It is a trend common to all sectors but with greater emphasis within sectors like WASH, Health, and Nutrition.

Changes in global standards: some trainings are driven by changes in concepts, technology, international protocols, global standards, concepts and strategies. This is very common in the health and nutrition sector.

12 The UNICEF Strategic Plan, 2014-2017. United Nations Children’s Fund, Executive Board Second regular session 2013. 3-6 September 2013. Item 4 of the provisional agenda. E/ICEF/2013/21. Pages 11-12. 13 UNICEF, 2013. Nigeria Country Programme Document 2014-2017. Page 8 14 UNICEF Nigeria Country Programme Document 2014-2017. Page 8.

15 All the sectors except the Child Protection follow the same approach to developing and delivering training activities. The Child Protection sector mostly works primarily at the state level unlike the other four sectors that has national coverage.

FINAL REPORT 19

© UNICEF

49. According to survey respondents, 65% indicated that they were consulted on their training needs prior to the training (Figure 3.1).

50. These consultative processes allow UNICEF to successfully engage with partners, to ensure that trainings offered respond to their needs for the implementation of programmes.

Figure 3.1 Participants’ needs assessment prior to the training

51. These training activities supported by UNICEF sectors can be relevant in developing or strengthening the capacity of State professional staff to deliver and to implement its programmes.

52. However, UNICEF does not conduct systematic capacity gaps analysis at the country level to identify priorities in terms of training needs, targeted participants and regions. As a result, the design and delivery of individual trainings by UNICEF in Nigeria has been reactive rather than proactive. Trainings run the risk of dispersion which undermines the relevance, effectiveness and efficiency of the training portfolio as a whole.

Finding 3: UNICEF Nigeria’s individual training activities respond to a certain extent to the needs of participants with opportunities for some possible improvements.

53. Surveyed training beneficiaries were overall satisfied with the training response to their needs. Indeed, 65% of survey respondents indicated that they were consulted on their training needs prior to the

training (Figure 3.2) and 85% felt that the training they participated in responded to needs they had before the training (Figure 3.3).

54. Respondents also found the training to be relevant to their jobs where 92% found that the UNICEF training they participated in was relevant to their work/main occupation (Figure 3.4) and 94 % found that the UNICEF training they participated in was needed to better do their job (Figure 3.5).

55. However, in the absence of M&E data to triangulate this self-reported data, there is limited indication on the extent to which the trainings actually helped beneficiaries better fulfill their role in supporting programme implementation and enhanced the partner organization’s performance overall.

56. Moreover, in terms of participants’ selection, according to UNICEF staff interviewed, the nomination of participants is generally done by State Implementing Partners. Survey results indicate that 53% of respondents were selected by their supervisor/local authority to attend the training while 38%

12%

15%

40%

26%

7%

I was consulted on my training needs prior to the training

Strongly disagree

Disagree

Agree

Strongly agree

Don't know

20 FINAL REPORT

© UNICEF

were selected by UNICEF (Figure 3.6). Due to the absence of data, the evaluation has not assessed the extent to which participants were qualified to attend the trainings.

57. As a result, the evaluation team observed that the current selection process does not allow UNICEF to ensure that training beneficiaries are strategically selected. The survey results show that 89% of participants were clear on why they were selected to attend the training. However, only 70% were clear about the purpose of the training before they attended, while 23% were not clear about its purpose (Figure 3.7). Survey results also show that while 56% indicated that the main reason that drove them to actually attend the UNICEF training was career enhancement, only 20% attended for their interest in the subject matter (Figure 3.8). In that sense, the relevance of the training recipients’ selection can be questionable, especially since trainings are not delivered by UNICEF for career enhancement purposes.

58. Consequently, the current training needs identification and participant selection processes give no insurance as to how strategic UNICEF is in responding to the training needs and how relevant and targeted its portfolio of trainings is in terms of selection of beneficiaries.

Figure 3.2 Respondents’ needs assessment Figure 3.3 Training response to respondents’ needs

12%

15%

40%

26%

7%

I was consulted on my training needs prior to the training

Stronglydisagree

Disagree

Agree

Strongly agree

Don't know

2%

7%

41%45%

5%

The UNICEF training I participated in responded to

needs I had before the training

Stronglydisagree

Disagree

Agree

Strongly agree

Don't know

FINAL REPORT 21

© UNICEF

Figure 3.4 Relevance to participants’ work/ main occupation

Figure 3.5 Training’s response to respondents needs to do their job better

Figure 3.6 Respondents’ selection to attend the UNICEF training

Figure 3.7 Clarity of Respondents about the purpose of the training prior to their attendance

2%

4%

37%

56%

1%

The UNICEF training I participated in was relevant to

my work/ main occupation

Stronglydisagree

Disagree

Agree

Strongly agree

Don't know

3%

2%

32%

62%

1%

The UNICEF training I participated in was needed to do my job better

Stronglydisagree

Disagree

Agree

Strongly agree

Don't know

1%

53%38%

1% 7%

Please indicate how you were selected to attend the UNICEF training

I self-enrolled

My supervisor/ local authority told me to attendthe trainingUNICEF invited me to participate in the training

I replaced someone else who could not attend

I don't know

7%

16%

41%

29%

7%

I was clear about the purpose of the training before I attended

Strongly disagree Disagree

Agree Strongly agree

Don't know

22 FINAL REPORT

© UNICEF

Figure 3.8 Reasons why respondents attended the UNICEF training

3.3 Effectiveness

59. We adopt the OECD DAC’s Glossary of Key Terms in Evaluation and Results-Based Management to define effectiveness as being “The extent to which the development intervention’s objectives were achieved, or are expected to be achieved, taking into account their relative importance.” 16

60. In this section, effectiveness is examined at two levels. First, we examine the extent to which the overall portfolio of UNICEF training is effective, namely the extent to which it has a clear strategy, is underpinned by a clear Theory of Change and includes appropriate feedback mechanisms allowing UNICEF decision-makers to make appropriate changes to the nature of the portfolio. Second, we review the effectiveness of the individual training activities, focusing specifically on training design, training effects (with the caveat that such effects are self-reported), and the existence of appropriate M&E systems.

16 Glossary of Evaluation and Results Based Management (RBM) Terms, OECD (2000). Page 22.

56%

2%

20%

20%

2%

Please indicate the main reason that drove you to actually attend the UNICEF training

Career enhancement

Networking

Had to attend because I was sent by my supervisor

Interest in the subject matter

Possibility of obtaining a certificate

FINAL REPORT 23

© UNICEF

Reviewing the actual quality of the training content is beyond the scope of this exercise and was not explored beyond anecdotal evidence provided by participants.

61. Overall, as will be described in the section, while the effectiveness of the training activities leads to some positive and less positive findings, the effectiveness of the portfolio as a whole is subject to improvement.

Finding 4: In the absence of a country training strategy and results framework, the overall effects of the training portfolio’s impact are jeopardized.

62. During inception and data-collection mission consultations, the evaluation noted the absence of a comprehensive national training strategy where trainings are strategically planned to respond to national training needs.

63. In that sense, the current approach to training design and delivery leads to a large diversity of ad-hoc uncoordinated trainings. For example, the list of trainings that the evaluation team captured through document review, interviews and survey results comprises a large diversity of training themes17.

64. A comprehensive national training strategy outlines the following elements18:

The overall strategic training objectives and the underlying Theory of Change;

The process by which UNICEF’s training is identified, and prioritized;

Tactics and actions required for the development, budgeting, and delivery of trainings;

A comprehensive results framework;

The process by which training standards are developed;

Follow-up and monitoring mechanisms to ensure training quality; and

The evaluation feedback loop to inform and report on the training performance overall.

65. Also, UNICEF Nigeria Country Programme’s monitoring and evaluation (M&E) system does not allow nor require the monitoring of training activities. The M&E Unit of UNICEF monitors the various sectors performance through independent monitoring of programme components; each section is responsible to monitor its own programme including the progress and result of its activities. Each sector is expected to submit an annual report for Results Assessment Modules (RAM) and Milestone Data for outputs. However, such reports are highly aggregated in the Annual Report and do not report on individual training activities.

66. Thus, in the absence of a robust results framework for its trainings, UNICEF is not assessing whether trainings have been delivered properly, and to whom. It does not have information on the outcomes of the trainings, and is not in a position to emit an informed judgment on the effectiveness of its training portfolio. Actually, this very evaluation is the first effort to perform such an assessment.

67. In the absence of national training capacity gaps assessments, strategic planning and prioritization of trainings, a comprehensive results framework, and feedback and learning loops, the trainings portfolio runs the risk of dilution of potential results.

17 Please refer to Appendix IX to read the full list of training themes captured through data collection

18 FAO, 2001. Asia-Pacific Forestry Commission - Regional training strategy: supporting the implementation of the code of practice for forest harvesting in Asia-Pacific - Food and Agricultural Organization of the United Nations, Regional Office for Asia and the Pacific. Bangkok, Thailand. (And) UNESCO (unknown year). Training of Trainers in Teacher Education for a Sustained Quality Education - External Evaluation Report.

24 FINAL REPORT

© UNICEF

Finding 5: From a design perspective and as Training of Trainer activities (ToT), the effectiveness of the training activities is mixed.

68. All sectors except the Child Protection sector adopt the same approach of delivering trainings through Training of Trainers (ToT). The training approach to Child Protection is somewhat different given the high technicality and the coverage of the programme: it mostly works at the state level, unlike the other four sectors that have national coverage. It consists mostly of direct delivery of training to those who need it to perform their duties, such as police officers, judges, juvenile courts, lawyers and child welfare officers.

69. To achieve an optimal multiplier effect, the actual model relies on the training of trainers, where trained trainers are expected to return to their respective State to develop and deliver training courses to support implementation. In that sense, trainings cascade down to the State and sometimes to the ward levels.

70. Benchmarking on international practices in terms of ToT design, Training of Trainers' design includes (or at least partially includes) the following features:19: 1) engaging with partners and conducting needs assessments; 2) including a focus on both content (in the case of UNICEF, this refers to the technical content) and on process (for instance, all notions related to group dynamics, principles of adult learning); 3) having a quality assurance system as the training cascades down; and 4) measuring results at four levels of the Kirkpatrick chain of training results.

71. Based on the evaluation's analysis, Table 3.1 summarizes the extent to which the design of UNICEF training complies with these features.

72. While Engagement with partners and needs assessment are covered in Finding 2, each of the following features is further discussed in subsequent findings (See Findings 6 to 8).

Table 3.1 Compliance of the actual ToT model with international practices

TRAINING OF TRAINERS DESIGN COMPLIANCE

Engagement with partners and needs assessment

Balance between technical training and training process (andragogy)

Quality assurance system

Measuring results at four levels

Legend: Not compliant; Somehow compliant; Compliant

19 Jean C. Murphy, Ed. D. and Carol O. Carson-Warner, Ed. D. (unknown date). Train-the-Trainer Manual - Chicago State University, Chicago, IL. And http://www.fao.org/docrep/W8088E/w8088e03.htm page consulted February 20th 2017. And UNDG Capacity Assessment User Guide, Feb 2008.

http://www.fao.org/docrep/W8088E/w8088e03.htm

FINAL REPORT 25

© UNICEF

Finding 6: The trainings do not balance technical training with delivery process. Training activities are designed with the dual purpose of transferring technical knowledge to participants and to enable them to, in turn, transfer their knowledge to others. From a design perspective, the technical content of the training is of good quality, but insufficient time is dedicated to building participants' training skills.

73. A ToT design should include a balanced transfer of learning on content (technical) and on process (andragogy).20 While the content component of a training allows participants to understand and master the technical material, the process component intends to prepare trainees (and trainers-to-be) to become effective trainers, as to the method and practice of teaching. According to survey results, the technical training is strong, whereas no attention is paid to enhancing the skills of participants in terms of training delivery.

Technical training

74. The review of the quality of the training technical modules was beyond the scope of this evaluation. As such, evidence provided as to the quality of the training relies solely on feedback obtained from participants.

75. This being said, training participants revealed that respondents were satisfied with the quality of the technical trainings received. Survey data reveals that 97% of respondents found the quality of the

20 http://www.fao.org/docrep/X5665E/x5665e04.htm - page consulted February 20th 2017.

http://www.fao.org/docrep/X5665E/x5665e04.htm

26 FINAL REPORT

© UNICEF

presentation and lecturing to be high, and agreed the content was presented in a way that was easy to understand21 (Figure 3.9).

76. Interview respondents also indicated that training materials, such as training guidelines, training manual existed–assertion verified by the evaluation team–and were distributed at the onset of each training session.

77. Ninety-six percent of survey respondents also confirmed the strong relevance of the training materials which they felt useful to support their learning. Similarly, 91% of respondents confirmed that training materials were adapted to their needs and prior knowledge (Figure 3.10) and 96% of respondents indicated that they still refer to the training materials as a useful source of information (Figure 3.11).

78. However, some concerns were shared regarding the level of complexity and timeliness of training guidelines and materials. Some of the training materials were reported to be complex and difficult to teach at the community level, which might affect the quality of trainings delivery by trained trainees at the state and ward levels. In that sense, the survey results revealed that 31 respondents recommended that improvements be made on training materials.

Figure 3.9 Quality of presentation and lecturing

21 Data disaggregation indicates that results do not vary significantly between the different sectors.

1%1%

40%

57%

1%

The quality of the presentation and lecturing was high and content was presented in a way that was easy to understand

Strongly disagree

Disagree

Agree

Strongly agree

Don't know

FINAL REPORT 27

© UNICEF

Figure 3.10 Usefulness of training material during the training sessions

Figure 3.11 Usefulness of training material after the training sessions

Training on Training Process

79. The assumption that, by having been exposed to a technical workshop, any given participant will be able, in turn, to deliver the same workshop to another audience is a weak assumption. Therefore, effective ToT design complements the technical training modules with some notions of a training process. This can include a wide range of topics, such as training content design, classroom management, effective training delivery, group dynamics, providing feedback to participants, all in all, a series of themes that allow any given trainer to deliver his/her content effectively.

80. As it stands, UNICEF Nigeria’s individual training activities limit themselves to delivering a solid technical component. The trainings do not include any modules on training process, or how to deliver trainings, nor does it provide training participants with the opportunity to practice their teaching skills prior to delivering trainings. Indeed, at the moment, the training is based on the assumption that participants will be able to deliver training after attending a technical workshop. This assumption could be true in some cases, but needs to be verified.

Finding 7: The current training design does not have a quality assurance mechanism, which represents a risk to training effectiveness.

81. Once the training of trainers completed, quality assurance becomes an important step in the design and delivery of ToTs. Currently, there is no framework ensuring that cascaded trainings delivered through the ToT approach remain of quality.

Guidance of trained trainers

82. A field-based trainer (Supervisor) who is the primary liaison between UNICEF and the future trainer is necessary to facilitate the future trainer’s transition into a competent trainer. This includes evaluating the trainer’s competence, providing support and clarifications on contents to the trainee, and submitting observations and recommending the trainee for readiness to serve as a trainer.

83. UNICEF provides additional support to training participants; 82% of survey respondents indicated that UNICEF carried out follow-up activities (contact, further support) after the training. However, there

1%1%

40%

57%

1%

Training materials (handouts, documentation) were useful and

assisted my learning

Stronglydisagree

Disagree

Agree

Strongly agree

0%2%

43%

53%

2%

I still refer to the training materials as a useful source of

information

Stronglydisagree

Disagree

Agree

Stronglyagree

28 FINAL REPORT

© UNICEF

is no evidence of follow-ups in terms of evaluating the trainer’s competence and readiness to serve as a trainer.

Updating knowledge

84. According to adult learning theory, people who train others remember 90 percent of the material they teach22. In the absence of a mechanism where trainers’ knowledge is updated, the training quality can degrade after only a few repeated trainings (from the national level to the state and ward levels). In that sense, of survey respondents, 45% agreed with the statement: “I have an overall feeling that, unless more training is provided, I will not retain my new knowledge/capacities” and 48% disagreed. Therefore, additional refresher trainings could be necessary to ensure an adequate delivery of trainings. Also, if any changes are brought to the modules, due to changes to the global standards in the subject matter, updating knowledge ensures proper transition of trainers to the new modules.

85. The current ToT approach does not proceed to any qualification test or ‘certification’ process of its trainees to identify the extent to which they can deliver trainings effectively, nor does it provide additional learning opportunities on the subject matter for the trainees to update their knowledge. In that sense, cascaded trainings run the risk of quality degradation.

Finding 8: Some efforts are made to monitor training participants’ learning at the field level. However, the monitoring of participants’ reaction, learning, changes in behaviour and impact is not systematic.

86. The programme sections supporting the training do not have complete information on the nature, type, duration and outcomes of training activities that have been conducted from 2014 to June 2016 - the period being evaluated. In the eventuality where the Evaluation Unit requests information from different sectors on the training activities, the availability of data depends on the willingness of the leadership of each sector to provide information. Data, if available, remains concentrated at the Field-Office level but somewhat uncoordinated and unsystematic23.

87. The table below summarizes the availability of information by Rick Davies Checklist.

22 http://www.urban.org/research/publication/assessing-train-trainer-model-evaluation-data-democracy-ii-project - Page consulted February 20th 2017. 23 A new Harmonized Approach to Cash Transfer (HACT) was introduced in 2014. It requires the submission of an Activity Report as part of requirements for financial reporting. The new approach proved to be challenging, where nearly all the Activity Reports were mostly descriptive and did not include any analysis or any feedback on challenges met and lessons learned on training design and delivery.

http://www.urban.org/research/publication/assessing-train-trainer-model-evaluation-data-democracy-ii-project

FINAL REPORT 29

© UNICEF

Table 3.2 Availability of Information: Rick Davies Checklist

INFORMATION AVAILABILITY QUESTIONS

ANSWERS

Is a complete set of documents available?

No.

Documentation is insufficient for the evaluation of most training, thematic sectors, evaluation questions, and Kirkpatrick levels. Additionally, documentation that is available is partly inconsistent in content and format across thematic areas and time.

Do baseline measures exist?

No (for training)/Yes (for CPD outputs relevant to training activities).

The evaluation found a limited number of pre- and post-tests carried out before and after training. However, these are not available for all training interventions.

At the UNICEF CPD output-level, baseline information is usually available.

Is data being collected for all the indicators?

No.

There are no specific set of indicators for training interventions, which are considered ‘activities’ in UNICEF’s Work Plans. Only in certain sectors (WASH, nutrition, and education) do individual training activities have indicators attached to them (almost always in terms of number of persons to be trained). Training interventions are usually expected to more broadly contribute to output-level indicators (for which data is usually available).

Is participant, beneficiary, and stakeholder data available?

Not systematically.

The evaluation collected some participant lists for the vast majority of training activities. However, such information was not systematically available either at the UNICEF NCO or at the Field Office level.

Is gender disaggregated data available?

Not systematically.

It may be possible to disaggregate training participants by men and women as some training participants’ lists keep track of this information. Documentation does not provide such gender-disaggregated data consistently.

If reviews or evaluations have been carried out, are the reports available?

No

No formal evaluation has previously been carried out on training investments as a whole. Reports are usually written at the level of the individual training activity; however, as with other documentation, the evaluation did not have access to these reports for most training activities.

Do existing M&E systems have the capacity to deliver, and to collect presently missing data in the future? (budget, management, capacity)

No.

A major limiting factor in the context of this evaluation is that no M&E system was built for training as such. The PME relies only on the information provided by section chiefs to track performance. The information is highly aggregated and often lumped together with other intervention implementation. As such, a comprehensive data base for training does not exist.

88. Evidence collected through document review and interviews suggests that there is a discrepancy between sectors in the systematization and depth of monitoring of training activities. The following table presents a summary of the data on trainings currently collected by sectors at the Field-Office level.

30 FINAL REPORT

© UNICEF

Table 3.3 Availability of M&E data on trainings at the field level, aggregated by sector

KIRKPATRICK LEVELS HEALTH NUTRITION EDUCATION CHILD

PROTECTION WASH

Reaction

Knowledge and skills

Change in behaviour

Results

Legend: Not monitored; Not systematically monitored; Systematically monitored

89. The evaluation observed that there had been some effort, however not systematic, to test training participants’ learning at all sectors, where 88% of survey respondents indicated that they were tested on their knowledge at the start and/or at the end of the training. In cases where trainings delivered captured the list of trainees and their affiliations, the level of attainment of training objectives, lessons learned, and participants’ feedback remain unavailable in most

Date post:	13-Feb-2021
Category:	Documents
Upload:	others
View:	4 times
Download:	0 times