US AID/Jordan Human Resources for Health in 2030 Midterm...

USAID/JORDAN HUMAN RESOURCES FOR HEALTH IN 2030 (HRH2030) MIDTERM ACTIVITY EVALUATION

January 2019

This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Alanna Shaikh, Raed Azmi, Hamouda Hanafi, Khaled Hasan, and Wisam Qarqash.

Cover photo: A health care provider at work. Credit: HRH2030.

USAID/JORDAN HUMAN RESOURCES FOR HEALTH IN 2030 (HRH2030) MIDTERM ACTIVITY EVALUATION

January 2019

USAID Contract No. AID-OAA-C-14-00067; Evaluation Assignment Number: 590

DISCLAIMER

The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government.

This document is available online at the GH Pro website at http://ghpro.dexisonline.com/reports- publications. Documents are also available at the Development Experience Clearinghouse (http://dec.usaid.gov). Additional information is available from:

Global Health Program Cycle Improvement Project

1331 Pennsylvania Avenue NW, Suite 300

Washington, DC 20006

Phone: (202) 625-9444

Fax: (202) 517-9181

http://ghpro.dexisonline.com/reports-publications

This document was submitted by GH Pro to the United States Agency for International Development under USAID Contract No. AID-OAA-C-14-00067.



http://dec.usaid.gov/


HRH2030 MIDTERM ACTIVITY EVALUATION / i

ABSTRACT This midterm formative performance evaluation of the USAID-funded Human Resources for Health 2030 (HRH2030) Activity addressed seven questions:

1. To what extent are human resources (HR) practices at the Ministry of Health improving and why?

2. To what extent are staff receiving training and using their newly acquired skills? Why or why not?

3. To what extent are capacity, knowledge management, transparency, and accountability in planning, managing, and retaining Jordan’s health workforce increasing? Why or why not?

4. To what extent is the Jordan Human Resources for Health 2030 (HRH2030) Activity influencing women’s management and leadership in the health workforce?

5. What are the next steps for institutionalizing the new law that requires continuous professional development for Jordan’s health workforce?

6. How can USAID best sustain the achievements and reforms accomplished under HRH2030 in Jordan?

7. What does the evaluation team recommend regarding any potential follow-on HRH activity? If a follow-on is necessary, should it be similar to HRH2030, or be combined with another existing USAID/Jordan activity, or take the form of direct support to the government of Jordan through the utilization of partner government systems?

The evaluation team found that HR practices were not improving overall, although some people were utilizing new HR tools. Capacity, knowledge management, transparency, and accountability are increasing slowly, but it is uncertain whether these gains are sustainable. HRH2030 interventions to influence women’s management and leadership have potential but have not been active long enough to show effects. Next steps for the relicensure bylaw include support to disseminate it and its accompanying instructions.

USAID can sustain HRH2030 achievements through continued support to the Women in Leadership Network, institutionalizing training, supporting the relicensure bylaw, and working with government partners beyond the Ministry of Health. Recommendations for a follow-on activity include supporting a body to implement the bylaw, combining service delivery and policy projects into a single activity, reducing the number of awards but implementing larger activities, and taking a multi-sectoral approach.

HRH2030 MIDTERM ACTIVITY EVALUATION / ii

ACKNOWLEDGMENTS The evaluation team acknowledges with gratitude the contribution of the esteemed heads of departments and high-level managers from the Ministry of Health, professional councils and associations, other governmental and nongovernmental entities, and international organizations. We would also like to thank the USAID implementers who participated in the evaluation. We are also grateful to staff at the Ministry of Health and its health facilities in Amman, Ramtha, and Balqa who participated in our focus group discussions and quantitative survey.

We wish to thank the HRH2030 Activity team in Amman for their full collaboration and responsiveness, and also express our sincere appreciation to the USAID team in Amman for the guidance and feedback they provided throughout the evaluation.

HRH2030 MIDTERM ACTIVITY EVALUATION / iii

CONTENTS Abstract ................................................................................................................................................................................. i

Acknowledgments ............................................................................................................................................................. ii

Acronyms............................................................................................................................................................................ iv

Executive Summary ........................................................................................................................................................... v

I. Introduction ..................................................................................................................................................................... 1

Evaluation Purpose ........................................................................................................................................................ 1

Evaluation Questions .................................................................................................................................................... 1

Limitations in the Evaluation Mandate ..................................................................................................................... 1

II. Project Background ...................................................................................................................................................... 2

III. Evaluation Methods and Limitations ........................................................................................................................ 3

Sampling ........................................................................................................................................................................... 4

Limitations....................................................................................................................................................................... 4

Data Analysis .................................................................................................................................................................. 4

IV. Findings........................................................................................................................................................................... 5

Evaluation Question 1 .................................................................................................................................................. 5

Evaluation Question 2 .................................................................................................................................................. 9

Evaluation Question 3 ................................................................................................................................................ 13





Crosscutting Findings ................................................................................................................................................. 21

V. Conclusions and Recommendations ...................................................................................................................... 23

Conclusions .................................................................................................................................................................. 23

Recommendations ...................................................................................................................................................... 24

Annex I. Scope of Work ................................................................................................................................................ 28

Annex II. Data Collection Instruments ....................................................................................................................... 47

Annex III. Documents Reviewed .................................................................................................................................. 70

Annex IV. Additional Feedback on Training .............................................................................................................. 73

Annex V. Statement of Differences ............................................................................................................................. 75

Annex VI. Disclosure of Any Conflicts of Interest .................................................................................................. 79

Annex VII. Summary Bios of Evaluation Team .......................................................................................................... 86

HRH2030 MIDTERM ACTIVITY EVALUATION / iv

ACRONYMS CPD Continuing professional development

CSB Civil Service Bureau

FGD Focus group discussion

HHC High Health Council

HML Health management and leadership

HR Human resources

HRD Human resources development

HRH Human resources for health

HRH2030 Human Resources for Health 2030

HRM Human resources management

HRMS Human resources management system

KII Key informant interview

KM Knowledge management

MOH Ministry of Health

USAID United States Agency for International Development

WHO World Health Organization

WISN Workload Indicators of Staffing Need

HRH2030 MIDTERM ACTIVITY EVALUATION / v

EXECUTIVE SUMMARY EVALUATION PURPOSE

As commissioned by USAID, this evaluation assessed the performance of the Human Resources for Health 2030 (HRH2030) Activity’s contribution toward a strengthened health workforce for better health services in Jordan. It was also designed as a formative performance evaluation to develop recommendations for the Activity’s remaining years, both to increase effectiveness and strengthen partnerships with government stakeholders to foster sustainability.

EVALUATION QUESTIONS

The evaluation addressed seven questions:

1. To what extent are human resources (HR) practices at the Ministry of Health (MOH) improving and why?


3. To what extent are capacity, knowledge management, transparency, and accountability in planning, managing, and retaining Jordan’s health workforce increasing? Why or why not?

4. To what extent is the Jordan HRH2030 Activity influencing women’s management and leadership in the health workforce?




ACTIVITY BACKGROUND

HRH2030 is a global cooperative agreement consisting of global and country-specific activities in multiple countries. Jordan was the first country to buy into the award and the first to undertake a midterm evaluation. Led by Chemonics International, HRH2030 consortium members in Jordan include the Palladium Group and the University Research Co. The Activity works with and through Jordan’s MOH, the High Health Council, and the Civil Service Bureau (CSB), as well as other professional and civic health organization stakeholders. It assists the MOH to enhance its HR practices, facility management, and leadership. It works with the High Health Council to strengthen national HRH governance and HRH data for decision-making, and collaborates with other national health stakeholders to improve health workers’ competencies by institutionalizing continuing professional development (CPD). Its interventions include training on health management and leadership, human resources development, and human resources management.

EVALUATION DESIGN, METHODS, AND LIMITATIONS

USAID conceived this evaluation as both a performance evaluation and a formative evaluation. A midterm performance evaluation usually serves two immediate purposes: decision-making and taking stock of initial lessons learned. These purposes are reflected in the first five evaluation questions, which have been addressed through the presentation of findings from all data sources listed in the

HRH2030 MIDTERM ACTIVITY EVALUATION / vi

methodology section, followed by conclusions and recommendations. Additional recommendations were also sought through Questions 6 and 7, focusing on USAID assistance beyond HRH2030. The findings for these questions reflect the opinions collected from the qualitative interviews and focus group discussions (FGDs), the analysis of collected data and existing documentation, and the views of the evaluation team.

This performance evaluation was unusual in that it sought to address questions not normally covered in the context of a performance evaluation. Questions 1-3 look at system-level change, while questions 2 and 4 are more traditional performance evaluation questions. Questions 5, 6, and 7 call for opinion and analysis from the evaluation team beyond the traditional scope of findings. Furthermore, the evaluation questions posed by USAID look at changes to the health system as a whole rather than specific effects of HRH2030.

The evaluation took a mixed-methods approach. Data collection consisted of a desk review, key informant interviews, FGDs, and a quantitative survey to gather information on the seven wide-ranging evaluation questions. The quantitative data was analyzed primarily using descriptive statistics. Whenever feasible, data were stratified by demographic characteristics, such as sex, age, and location.

FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS

Findings

Evaluation Question 1: To what extent are HR practices at the MOH improving and why?

HRH2030 reports provide examples of how training participants introduced changes at their health directorates using learning acquired from the Activity. There was progress toward harmonization of the data in the Human Resources Management System (HRMS) between the MOH and the CSB. The MOH made strides toward strengthening its management information system staff by adding new recruits, and it completed the cleaning of personnel data for migration to the HRMS. The HRH2030 team identified progress in support of the Workload Indicators of Staffing Need (WISN) tool as a major achievement. It has the potential to provide new information to support evidence-based staffing decisions, and the MOH has expressed demand for WISN data.

The Activity’s inputs, such as training, facilitating policy development, and drafting various tools used in HR management, will not lead to change on their own; they are necessary but not sufficient for change. Through the survey and FGDs, the evaluation team found that only those managers colleagues or staff regarded as intrinsically motivated attempted new management approaches or used new tools. Cultural or systemic support for implementing new ideas does not exist. The MOH’s work culture and systems do not support new behaviors, such as improved performance, innovation, and initiative. The MOH and CSB compensation systems do not recognize or reward innovation or increased performance.

Evaluation Question 2: To what extent are staff receiving training and using their newly acquired skills? Why or why not?

To date, HRH2030 has trained between 300 and 400 people. It is difficult to determine the exact number, as some people attended more than one training event. Participants trained in HR management, HR development, and health management and leadership seem to be using their new knowledge and skills only to a limited degree. According to FGDs and the quantitative survey, these people are not supported in using those skills, and their ability to apply them in practice is limited by uncertainty and lack of time.

The FGDs with staff whose supervisors attended HRH training indicated that there have been improvements in the supervisors’ behavior and management. There are signs of leadership, cooperation, and respect for time, which go beyond supervision. Furthermore, there is a difference in the health

HRH2030 MIDTERM ACTIVITY EVALUATION / vii

management and leadership trainee’s behavior in areas such as sharing information, day-to-day management, showing flexibility, conducting more meetings with agendas and minutes, and commitment to regular attendance.

Evaluation Question 3: To what extent are capacity, knowledge management, transparency, and accountability in planning, managing, and retaining Jordan’s health workforce increasing? Why or why not?

There is no evidence of planning for the MOH health workforce other than what is in the Ministry’s strategy document, which expresses overall staffing targets without a clear plan to achieve them. Signs of improvement can be found in the number of policies and procedures developed; HRMS improvements; interventions to improve employee orientation; newly acquired employee skills; development and updates of job descriptions; attempts to determine staffing needs and address uneven distribution of personnel; attempts to improve women’s participation in management; and improvements in leadership through training. However, FGDs and key informant interviews also frequently noted the continuation of old hiring practices, improper selection of training participants, lack of transparency in employee appraisals, and lack of fairness in assessing employee performance and determining promotions and incentives. FGDs and key informant interviews also indicated an absence of transparency and accountability. Management decisions are being made not only without consultation but often at the wrong level, usually much higher than needed. Decisions are made according to inappropriate criteria such as personal interests, wasta (a corrupting influence), gender discrimination, and favoritism.

Evaluation Question 4: To what extent is the Jordan HRH2030 Activity influencing women’s management and leadership in the health workforce?

The evaluation team collected data in September 2018, when the Activity had been in place for three years and had been actively working with the MOH for two years. HRH2030 launched its first research on barriers to women’s leadership in June 2018 and established a forum for women in the health sector in mid-September 2018. It is unrealistic to expect culture change that leads to additional support for women in leadership positions in just two years, especially when the goal is widespread change in HRH, not only the advancement of women. At the time of this evaluation, therefore, HRH2030 had not had any influence on women’s management and leadership in the health workforce, though this appears to be due to the Activity’s timeline rather than the quality or appropriateness of its interventions.

Evaluation Question 5: What are the next steps for institutionalizing the new law that requires CPD for Jordan’s health workforce?

Dissemination: The relicensure bylaw—and instructions for its implementation, once they have been developed—need to be widely promulgated, beyond the usual approach of simply publishing them in the government Gazette.

Development of CPD capacity for remote areas: A government entity needs to develop CPD opportunities for providers who cannot travel to major cities or pay for expensive programs. This could be Web-based education, teleconferences, or travelling training programs that visit each governorate.

Monitoring and registration: The MOH, as the licensor of all health professionals, needs to have the capacity, with its affiliated health professional councils, to track which health care providers have met their CPD requirements.

Coordination: Major providers of CPD are likely to include, as per the draft instruction, the MOH, Royal Medical Services, the Jordan Medical Council, the Jordan Nursing Council, health professional associations, and public and private teaching hospitals. Coordination among these entities, especially for online training, through a recognized institutional arrangement will improve the overall quality of CPD.

HRH2030 MIDTERM ACTIVITY EVALUATION / viii

Evaluation Question 6: How can USAID best sustain the achievements and reforms accomplished under HRH2030 in Jordan?

Support the relicensure bylaw: USAID could advocate for the bylaw and work to ensure that appropriate implementation structures are put in place for its enforcement. It could also provide technical support in the areas of dissemination, capacity development, monitoring and registration, and coordination.

Engage the CSB: The major HR functions for government employees sit with the CSB, which is responsible for setting regulations for hiring, compensating, promoting, retirement, and monitoring the performance of all civil servants, including those working in MOH. USAID could provide support to the CSB to revise the civil service bylaws, including the articles related to promotion and training.

Support the Women in Leadership Network to support women leaders: Although HRH2030 is widely supported and stakeholders believe it has a lot of potential, the Network will be only two years old when the Activity ends. It is unlikely that it will be sustainable at that time. USAID could continue support to it until it grows into a self-sustaining entity.

Evaluation Question 7: What does the evaluation team recommend regarding any potential follow-on HRH activity? If a follow-on is necessary, should it be similar to HRH2030, or be combined with another existing USAID/Jordan activity, or take the form of direct support to the government of Jordan through the utilization of partner government systems?

USAID should consider reducing the number of awards but implementing larger activities in the period following HRH2030. It should also consider combining service delivery and infrastructure projects with interventions that are more challenging to stakeholders, including HRH interventions. This would allow the more challenging interventions to leverage the relationships and goodwill created by infrastructure activities. Last, USAID should consider a multisectoral approach to HRH that includes the Ministries of Finance, Higher Education, and Public Sector Development.

Conclusions

HRH2030 is concerned with improving HR practices. Long-term improvement will require scaling up the best practices it has initiated. However, Activity documents do not mention scaling up; furthermore, scale-up is not possible using approaches that rely solely on the Activity’s own resources and staff without engaging the capacity and resources of the host country.

One major challenge is that knowledge about HRH2030 and its interventions is poorly disseminated and very scarce. The absence of a clear strategy for communication within the MOH at central and directorate levels and with stakeholders does not foster transparency, participation, and accountability. The lack of communication and advocacy is detrimental to participation and transparency, to the visibility of MOH achievements, and to garnering support for change and new policies. It perpetuates the current perception that little or nothing has changed for the better at the MOH.

Recommendations

• Develop an Activity-level sustainability plan to ensure interventions have a long-term effect: HRH2030 currently addresses sustainability only at the intervention level.

• Seek high-level buy-in at MOH for interventions and focus on mid-level officials: HRH2030 should increase communication and advocacy efforts to disseminate products such as the relicensure bylaw and the National HRH Strategy.

• Prioritize local expertise and institutions whenever possible: HRH2030 should make every effort for more direct engagement with Jordan’s own resources. Due to the challenges of contracting and

HRH2030 MIDTERM ACTIVITY EVALUATION / ix

deadlines, HRH2030 has been unable to move beyond hiring local consultants into contracting local institutions. However, working with these institutions would help to build national capacity on HRH topics and support long-term impact.

HRH2030 MIDTERM ACTIVITY EVALUATION / 1

I. INTRODUCTION EVALUATION PURPOSE

As commissioned by USAID, this evaluation assessed the performance of the Human Resources for Health 2030 (HRH2030) Activity’s contribution toward a strengthened health workforce for better health services. It was also designed as a formative performance evaluation to develop recommendations for the Activity’s remaining years, both to increase effectiveness and strengthen partnerships with government stakeholders to foster sustainability.

The primary audience for this analysis is the leadership of USAID/Jordan, including the leadership of the Population and Family Health Office. Secondary audiences include USAID/Washington and other USAID missions that may be interested in the Activity, the implementing partners (Chemonics and others), the government of Jordan, and other stakeholders.

EVALUATION QUESTIONS

The evaluation addressed seven questions:

1. To what extent are human resources (HR) practices at the Ministry of Health (MOH) improving and why?


3. To what extent are capacity, knowledge management (KM), transparency, and accountability in planning, managing, and retaining Jordan’s health workforce increasing? Why or why not?

4. To what extent is the HRH2030 Activity influencing women’s management and leadership in the health workforce?




LIMITATIONS IN THE EVALUATION MANDATE

The evaluation was conducted at the midterm of HRH2030 and did not include an examination of its inputs and outputs. While the project’s outputs were available through the periodic reports, the data collection tools did not seek to measure the Activity’s efforts and inputs, or the speed and cost-effectiveness of its execution, and there was no direct intent to measure levels of achievements against pre-stated midterm objectives. Although some of the evaluation questions speak of changes in the MOH human resources (HR) situation and application of training results, this evaluation was not expected to measure changes in the indicators identified in the HRH2030 Activity Monitoring, Evaluation, and Learning Plan (AMELP), but primarily aims to provide qualitative information and insights to respond to the evaluation questions, including the development of recommendations based on the findings.


II. PROJECT BACKGROUND HRH2030 is a global cooperative agreement consisting of global (or “core”) activities and country-specific activities (or “buy-ins”) in multiple countries. Jordan was the first country to buy into the award and the first to undertake a midterm evaluation. Led by Chemonics International, consortium members in Jordan include the Palladium Group and the University Research Co. The Activity works with and through Jordan’s MOH, the High Health Council (HHC), and the Civil Service Bureau (CSB), as well as other professional and civic health organization stakeholders.

HRH2030 activities build on the decades-long partnership between the United States and Jordan in the health sector, which has contributed to significant and tangible improvements in infant and child mortality rates and health care service delivery. However, Jordan’s rapidly growing population, an influx of Syrian refugees, and the decentralization of the health care system have intensified challenges for health practitioners and managers alike. Nurses and other professionals face increased workloads, unclear career development pathways, and strained information systems. Addressing these challenges is key to fostering a stable and resilient health workforce.

Under the Mission’s Development Objective 3 (Social Sector Quality Improved), Intermediate Result 1 (Health Status Improved), USAID has emphasized its commitment to improving health sector outcomes and resilience through health systems strengthening. Through HRH2030, USAID seeks to help Jordan strengthen its health workforce through interventions that advance three key results:

• Result 1: Improved HR practices at the MOH

• Result 2: Improved competency of the health workforce

• Result 3: Strengthened national HRH governance

The Activity’s theory of change, established by the Mission, is that if the HR practices and workforce competency are improved and national HRH governance is strengthened to provide better access and quality, then, as a result of a strengthened health workforce, health services will be improved for the Jordanian population. In the HRH2030 results framework, sub-results are aligned under the three key results they advance. For example, improved HR practices at the MOH (Result 1) requires having trained HR individuals (Sub-result 1.2) who are capable of developing and/or improving the HR management and development systems (Sub-result 1.1) that cover HR functions, including recruitment, orientation, development, planning, motivation, and succession planning.

Interventions under each sub-result were based on the review of outcomes in the Activity’s first three years as well as stakeholder priorities. This included integrating research topics into the applicable interventions rather than having stand-alone research interventions. Last, crosscutting themes in the results framework include the need to address fragmentation in the health sector, decentralization efforts across the public sector, gender, and communications to raise awareness on HRH issues.

HRH2030 began in Jordan on December 31, 2015; it had been active for almost three years at the time of evaluation data collection. However, due to a major reorganization during Year 1, most interventions had been in place for two years or less when data collection began. Interventions specific to women’s leadership in health had been in place for less than six months at the time of the evaluation.


III. EVALUATION METHODS AND LIMITATIONS A midterm performance evaluation usually serves two immediate purposes: decision-making and taking stock of initial lessons learned. These purposes are reflected in the first five evaluation questions, which have been addressed through the presentation of findings from all data sources listed in the methodology section, followed by conclusions and recommendations.

Additional recommendations were also sought through questions 6 and 7, focusing on USAID assistance beyond HRH2030. The findings for these questions reflect opinions collected from the qualitative key informant interviews (KIIs) and focus group discussions (FGDs), analysis of collected data and existing documentation, and the views of the evaluation team.

The second element of this evaluation’s formative purpose, “strengthen partnerships with government stakeholders to foster sustainability,” views relationships with stakeholders as a basis for increasing sustainability. The contacts developed for data collection includes most, if not all, stakeholders from the evaluation scope of work. The evaluation team also considered additional relevant stakeholders.

The evaluation team sought to pinpoint perceptions of ownership and indications of stakeholder participation as desirable key features in the Activity’s relationships with its stakeholders. The team analyzed the approaches HRH2030 used to develop its plans and products, and looked at what legacy the Activity might leave for Jordan in terms of tools and systems.

The evaluation team developed a total of six data collection tools—KII questionnaires and five FGD guides—which it shared with USAID and HRH2030 and had translated into Arabic. The team also developed a quantitative survey questionnaire. The table below lists the data collection tools, all of which are included in Annex II.

List of Data Collection Tools Prepared

# Target Population Type of Tool 1 FGD guide for health service providers FGD guide

2 FGD guide for directorate mid-level supervisors and for facility managers FGD guide

3 FGD guide for women in management positions FGD guide

4 FGD for HRH staff trained by HRH2030 in human resources management/human resources development and health management and leadership FGD guide

5 FGD guide for women with non-supervisory role FGD guide

6 KII guide for USAID partners other than HRH2030 implementing partners (e.g., Jordan Health Finance and Governance Activity; Jordanian Communications, Advocacy, and Policy Activity; Health Service Delivery Activity; and Takamol Activity)

KII guide

7 KII of government officials other than from MOH (Jordan Medical Council, Jordan Nursing Council, CSB, HHC) KII guide

8 KII World Health Organization (WHO) and international organizations KII guide

9 KII Senior MOH officials/managers KII guide

10 KII for HRH2030 teams KII guide

11 KII for private and civil society organizations (e.g., Jordan Medical Association, Nursing and Midwifery Association, Private Hospitals Association, Patient Protection Coalition, and Al Makassed charity hospital)

KII guide

12 Standard introduction and consent Intro to KIIs


# Target Population Type of Tool 13 Standard introduction and consent Intro to FGDs

14 Standard introduction and consent Intro to Survey

15 HRH2030 trainees and facilities staff not trained by HRH2030 Questionnaire SAMPLING

As this is a performance evaluation, not an impact evaluation, it does not seek generalizable results. As such, it used a mixture of convenience and snowball sampling during data collection. Key informants (n=31) were selected for interviews based on stakeholder recommendations and desk review data. The evaluation team conducted 13 FGDs, each with an average of six participants. FGD participants were chosen according to selection criteria the team developed, which ensured participation by females and a range of employees in the MOH system. For the quantitative survey, the evaluation sought to reach every person who had received training from HRH2030, as well as people in comparable roles who had not received training. In all, 259 people responded to the survey: 48 online and 211 in person.

LIMITATIONS

Some HRH2030 interventions, such as the second cohort of health management and leadership (HML) training and the Women’s Forum, had only recently begun at the time of data collection. Therefore, the opportunity to look at results and achievements was limited. Contact information was not available for all personnel working under the supervisors who attended HML training. Therefore, the evaluation team limited the data collection to the staff in Amman, Balqa, Karak, and Ramtha.

The e-survey encountered a distribution problem during the testing phase: The invitation to take the survey via SurveyMonkey went into junk/spam folders in recipients’ email accounts. Therefore, the evaluation team decided to share the link to the online survey on HRH2030’s Facebook groups, which were created to communicate with training participants. Therefore, the respondents to this survey were people who had been trained and who choose to be part of the training alumni Facebook group. For the most part, this group has positive feelings about the training and/or the group of fellow trainees. It is likely that this led to a bias in favor of training.

One disadvantage of an FGD is that people might be uncomfortable expressing candid opinions in front of their peers or may hesitate to express thoughts that differ from the group as a whole. Some key informants were more forthcoming than others. This may have related to informants’ perception of the confidentiality promise the evaluation team made.

Triangulation across data sources minimizes the effect of these limitations. When concerns regarding the depth of evidence or ability to interpret results affected the evaluation findings, these limitations have been noted in the text.

DATA ANALYSIS

Qualitative data was coded using R QDA to identify prevailing themes and frequency of core concepts. Quantitative data was analyzed with Excel, using data that was disaggregated by gender, length of time with MOH, participation in HRH2030 training, and other parameters. While the data set was not large enough to require serious R analysis, the use of coding software allowed faster and more consistent results than hand-coding.


IV. FINDINGS EVALUATION QUESTION 1

To what extent are HR practices at the MOH improving and why?

HR practices at the MOH as presented in the USAID/HRH2030 logic model1 refer to human resources management (HRM) and human resources development (HRD) systems on one hand, and the capacity of the MOH staff on the other. Key practices in these areas are planning, training and development, motivation systems, career paths, succession planning, and human resources management systems (HRMS). In the HRH2030 Performance Indicators Reference Sheets (PIRS),2 “HRH management best practices refer to the HRM and HRD best practices that fall under the following areas: HRM/HRD capacity, personnel policy and practice, HRM and HRD data, performance management and training and staff capacity. Improvement means the practice under the related area or sub-area shows progress and moves to a higher stage in HRM/HRD assessment matrix.”

Improvements in the MOH practices were not directly measured or necessarily observed in the evaluation; instead, they were assessed through the FGDs, KIIs, and the quantitative survey. Some respondents may not have been aware of the precise definitions of HRH practices when they expressed their views. They may also have been influenced by other aspects of the MOH HR field, such as policies or provision of health services. To compensate for this, these perceptions were triangulated with the documents from the desk review, which looked at the juridical and structural underpinnings of the MOH and CSB. Collecting hard data on HR practice improvement, such as percentage of employees receiving performance reviews, rates of absence, and percentages of high and low performing employees, was beyond the capacity of this evaluation.

It is appropriate in a midterm evaluation to look at possible interventions aimed at producing the desired changes and see to what extent such interventions are taking place, as an indication that a project is on the right track. Interventions listed in the results framework include an operational strategy for personnel affairs and HRD directorate; HR policies and procedures; new employee orientation; job levels linked to competency-based job descriptions; performance management; staff training needs and plans; harmonizing HRMS data; Workload Indicators of Staffing Need (WISN) data utilization; and HRD and HRM training for MOH staff.3

Last, Evaluation Question 1 does not specify that the improvements being sought are related to what HRH2030 has introduced or provided. The question addresses improvements within the MOH regardless of the source.

Results: Improvements in HR and adoption of new practices

The HRH2030 reports4 provide examples of how training participants introduced changes at their health directorates using learning acquired from training. They cite training workshops organized at the directorate level on topics such as teamwork in the Amman directorate; infection control in Petra; washing and hygiene for nurses in Karak, Mafraq, and Balqa health directorates; and a work restructuring plan in Jerash. Other topics include the development of a diversity and inclusion policy and procedure in Madaba, Karak, and Aqaba Health directorates, and the development and presentation of a work ethics implementation and reinforcement plan in the Maan directorate and at the MOH central level.

1 USAID/HRH2030 Activity Monitoring Evaluation and Learning Plan version #1. 2 USAID/HRH2030 Performance Indicator Reference Sheets, final. 3 Results Framework for HRH2030. 4 HRH2030 Quarterly Report, third quarter 2018.


In Quarter 3 of fiscal year 2018, HRH2030 reported that Human Resources Planning and Personnel Affairs had consolidated its action plan and the monitoring and evaluation plan after MOH restructuring. Nine new or revised HR policies and procedures were completed, as were new employee orientation tools. Approximately 110 competency-based job descriptions were developed, and WISN data collection was completed in seven health directorates, reaching 55 percent of the 622 of Jordan’s MOH health centers.

The MOH and the CSB made progress toward harmonizing the data within the HRMS. The MOH made strides toward strengthening its management information system staff by adding new recruits and completed the cleaning of personnel data for migration to the HRMS. As reported in KIIs, a significant part of the work on the harmonization of HRMS data was accomplished because HRH2030 embedded its information systems and data technical advisor part time at the MOH offices for three months to drive the process.

The HRH2030 team said progress in support of WISN was a major achievement. It has the potential to provide new information in support of evidence-based staffing decisions, and the MOH has expressed demand for WISN data. However, successful implementation of WISN faces three challenges. First, few facility-level staff understand its purpose, as was observed in the FGDs at the governorate level. Next, the consensus among HRH2030, other USAID implementers, and MOH staff is that WISN produces only preliminary data that are not reliable and cannot be used in staffing decisions. This is normal in a WISN system at the current state of implementation. Third, there is no guarantee that WISN, once its data is deemed reliable, will actually be used to determine staffing decisions.

Compensation systems: The MOH’s work culture and systems do not support new behaviors, such as improved performance, innovation, and initiative. The MOH and the CSB compensation systems do not have recognition or rewards for innovation or increased performance. The survey found that 27 percent of staff not receiving training from HRH2030 were satisfied with the level of motivation they receive at the MOH, whereas 51 percent in the group trained by HRH2030 were satisfied. This seems to indicate that receiving HRH230 training increased employee satisfaction at work.

Lack of recognition for administrative and support staff: Clinical workers are part of organized medical sectors. Nurses and midwives, medical doctors, dentists, lab technicians, and pharmacists are all regulated by official bodies and supported by their respective professional associations. However, FGD participants did not perceive support staff (e.g., administrative/financial staff and others who are not part of the clinical professions) as belonging to a profession, and said these workers do not have professional development opportunities. The only exception was accountants, who respondents said always received good orientation and training. As one service provider stated, “Accountants go to Amman for one week of training after they are appointed, but we don’t. I have worked for 18 years and did not receive training.” One result of the lack of recognition for administrative staff is that they are considered interchangeable and can frequently be moved from section to section. For example, one FGD discussion included two people who were trained in HRM and then moved to a job in medical records.

Analysis: Why are there improvements or why not?

There is a striking contrast in the views expressed in the HRH2030 reports and the perceptions of MOH staff and stakeholders. One major issue is that knowledge about HRH2030 interventions is very scarce and poorly disseminated. While close counterparts working with the Activity may be aware of specific achievements or progress, the majority of MOH officials—including heads of departments close to the HRH field—are not familiar with the changes. Stakeholder perceptions that there is little or no improvement can be attributed to a lack of visible changes on the ground, but are compounded by the fact that very little information has been shared about such improvements or achievements. According to one FGD participant, “The recent performance evaluation rules and regulations are not activated at


the health directorate levels. On the ground, there is a challenge related to policy implementation.” One key stakeholder said

“There is no improvement on the leadership at the MOH. It is personality-driven. [There are] no policies or approaches to supporting leadership. Right now, leadership is very bad; there is no succession planning even as so many people are forced to retire by CSB. Some mid-level managers are very competent, but it’s not because of support.”

A senior MOH official said, “International assistance is trying to implant a new culture of institutions, whereas we continue to have a culture of individuals. From recruitment to motivation, promotions, and compensation, the HR situation is worse than before.”

Even when major changes are decided, such as a new strategy (HRH) or a new bylaw (relicensure), information dissemination is poor prior to and during development and remains poor after adoption. This may relate to HRH2030’s approach of supporting MOH rather than engaging in direct intervention. If MOH is the entity enacting a policy or strategy, then it is also responsible for communication around that policy. In practice, however, it appears that communication either does not take place at all or is very limited in scope.

The first year for HRH2030 was characterized by staff changes and work plan revisions and redesigns. Functionally, at the time of the evaluation, the Activity had been active for only two years, which is too early to see results from a policy project. The improvements cited in HRH2030 reports cover most of the practices it targeted, such as planning, training and development, HRMS personnel policy and practice, HRM and HRD data, performance management, and training and staff capacity. However, it is too soon to determine if these improvements will lead to the long-term systemic change required for impact. The evaluation team saw no visible interventions or changes in motivation systems, career paths, and succession planning. These changes would have to come from working with the CSB, which HRH2030 has not done beyond its work for HRMS.

The inputs of HRH2030, such as providing training, facilitating policy development, and drafting tools used in HR management, will not lead to change on their own. They are necessary but not sufficient for change. The evaluation team found in the survey and FGDs that only managers whom colleagues or staff regard as intrinsically motivated will attempt new management approaches or use new tools. There is currently no cultural or systemic support for implementing new ideas. “People are motivated to work less, not more,” one key stakeholder said, “because when you work more you risk doing something you will be punished for.” An FGD participant said, “We are overloaded, so there is no time to apply what we have learned. Not all senior managers have interest in what we learned,” while another added, “We participated in training on HRM and it was great, but there is no support at our work. We held a meeting about performance evaluation and we wanted to see how we can have a human resources program and management.”

The survey results showed that only 47.5 percent of respondents received meaningful recognition for doing good work, and only 12.5 percent believed that the MOH recognizes good job performance.

Although review periodic reports and lists of achievements confirmed that HRH2030 introduced new practices, many respondents felt the MOH adopted the policies and interventions the Activity wanted to implement, not ones that actually addressed the Ministry’s needs. The gender component was among the interventions often mentioned as having been imposed from above, although others were also seen as being driven and owned by HRH2030, not the MOH (e.g., the relicensing bylaw and the National HRH Strategy). According to a key stakeholder, “HRH2030 prepared the [continuing professional development] bylaw and pushed it through the HHC despite the various issues and concerns raised about some of its items.”


As Evaluation Question 1 asks whether those changes at the MOH, were driven by the Activity, it is appropriate to address one major obstacle to systemic change that at least three key stakeholders outside of the Ministry mentioned. Wasta, or corrupting influence, is ingrained in the culture of MOH. Two-thirds of FGD participants agreed wasta was problematic, saying it affected all levels of the Ministry. They believe staff will not be promoted or selected for training without wasta. The survey results showed that only 23.8 percent of staff believed that recruitment, promotion, and learning opportunities were fair. According to the Jordan Country Profile (2015), corruption in Jordan is manifested mainly as favoritism, nepotism, or bribery. Favoritism in recruitment, hiring, transfer and promotion, and training opportunities, as well as the absence of justice and transparency, were perceived as the most prevalent areas of corruption practiced by health care organizers.5 Wasta is a key source of widespread mistrust and a barrier that prevents people from investing in systemic change.

Wasta is not the only challenge to systemic change. The scope of the training delivered by HRH2030 is too limited to achieve a “critical mass” at the facility or directorate levels, as the Activity was not designed to include building national and local training capacity and cascading training. This and other aspects of training will be further explored in the next section.

Leadership is another challenge to driving change. The frequent turnover at the higher levels of the MOH has forced HRH2030 to keep up with shifting priorities at the Ministry, and to constantly rebuild and renew the understanding and commitment to its interventions among officials. Experienced and knowledgeable key informants shared their view that the success of any intervention relies on engaging middle management and, especially, top-level management, including the secretary general (who has changed four times in the past three years) and the minister. Although HRH2030’s theory of change requires change to bubble upward to engage higher levels, the culture of MOH and the country does not appear to support this. As one FGD participant explained, “A previous minister had decided to associate the second and third tier of professionals in decision-making as a step toward improving succession planning, but the minister changed and we got back to the old practices again.”

The upshot of these challenges is that the work environment has not changed and does not easily accommodate new practices. Information-sharing is limited among personnel. Only 51 percent of respondents trained by HRH2030 and 39.5 percent of other respondents said their supervisors answered the questions they asked, while in FGDs there was talk about “work secrets” and how supervisors felt they retained control by keeping important information to themselves.

Most respondents in the sample did not feel that the MOH’s financial incentive system encouraged innovation or high performance. Participants in both KIIs and FGDs explained that MOH was bound by civil service standards for promotion and incentives, which are awarded based on years of experience and annual assessments. Some respondents also felt that incentives were provided on a non-transparent basis. The 2015 national HRH report 15 describes the system as follows: “The performance appraisal process in MOH and other public sectors is based on the evaluation of the overall behavior of the employee and his commitment to the official working hours. Incentives are not linked to actual performance.”6

Personnel expressed a low level of satisfaction with their compensation: only 18.2 percent of HRH2030 trainees and 8.6 percent of others believed they were receiving a fair market compensation, and only 18 percent of HRH trainees and 11 percent of others said they were satisfied with their salary and benefits. Despite the low compensation, some respondents said they moved from the private sector to MOH due to the job security those positions offered.

5 Musa Ajlouni. “Integrity and Corruption in the Health Sector in Jordan.” International Business Research; Vol. 10. 6 HRH 2015 report, Jordan High Health Council.


Furthermore, MOH may lack the basic infrastructure to support good HR practices. Only 51 percent of Activity trainees and 28 percent of others reported satisfactory work conditions. Seventy percent of the HRH trained staff had access to the internet at work versus just 33.5 percent of the others. In FGDs with support staff in the directorates, the majority said they did not have computers at work. Some also mentioned how difficult it was to work in very hot weather without any air conditioning.

EVALUATION QUESTION 2

To what extent are staff receiving training and using their newly acquired skills? Why or why not?

Training

Training from HRH2030: The Activity’s reporting provides detailed information regarding the numbers of MOH staff from the central level, the directorate level, and Ministry hospitals who have received training in HML, HRM, and HRD. Training in HML was meant to start in Year 2, after preparatory work that included a training needs assessment and requesting a list of 40 participants in March in order to hold two training modules. However, HRH2030 received a list of 67 participants in September and the training was delayed until Year 3.

In Year 3, and until the end of Quarter 3 of fiscal year 2018, between 56 and 62 participants attended the first eight training modules, at the rate of one day each month. At that point, HRH2030 reported the result that “three Health Management and Leadership participants were promoted and received positions with increased responsibilities.” The Activity then started the second cohort of the training course in Quarter 4, benefiting 50 participants.7

The training report for Year 1 shows that nine workshops were held benefiting a total of 139 participants, of whom 59 (43 percent) were female.8 The workshops were held in Amman; each lasted two days, with participants from outside the capital accounting for 57 to 67 percent of all trainees. Topics included WISN, “Supportive Supervision Training of Trainers,” and “Basics of Supportive Supervision Training.”

In Year 2, trainees traveled to Amman from governorates in the North (Irbid, Mafraq, Ramtha, Jarash, and Ajlun), Central Jordan (Amman, Madaba, Balqa, and Zarqa), and Karak. In the southern health directorates of Petra, Maan, Tafieleh, and Aqaba, workshops were held in Aqaba and Petra. This HRD/HRM training was presented in seven modules, with an average of 50 participants at the Amman workshops and 20 participants in the Petra or Aqaba workshops.

In Year 2, up to 55 MOH administrative staff attended two-day workshops on two occasions to develop the Administrative Affairs Administration Strategic Operational Plan for 2018-2020. Another activity targeting MOH personnel was an orientation for 20 health directors to introduce them to the HRH2030 Activity. Last, there were two events targeting the National HRH Observatory, which was formed to to be a primary source of reliable HRH data to inform policy development and decision-making: one about data collection and one entitled “Strategic Planning, Observatory Assessment and Annual Report Dissemination” (although the latter focused on the National HRH Strategy). In October 2017, at the request of the MOH, up to 89 hospital staff started—and continued—receiving training in HRM/HRD at a monthly one-day module. There was also a presentation on WISN so all health directors were oriented about the system and what it could do to help them to determine future staffing needs.

7 HRH2030 HML 2 Participants database (Excel table). 8 Year 1 training report, USAID HRH2030 Year 1 Annual Report. The report gave percentages of women in each of three training workshops; the actual numbers were calculated for this report.


Informant feedback about HRH2030 training interventions: Selection criteria were disregarded in some cases, contributing to perceptions of lack of transparency and the continued influence of wasta. According to feedback from the FGDs, a few participants may not have needed the training as they already had enough experience and previous training. FGD participants indicated that some MOH staff managed to be at every training.

The FGD with HML training participants showed consensus that participants did not receive feedback about the assignments they completed. Furthermore, two-thirds felt HRH2030 leadership did not hear their feedback, indicated that some assignments and tools were irrelevant to their work, complained about the performance of the training facilitator, and said they were “treated like kids.” FGD participants also expressed discontent with the length of the training and noted that some training was not relevant to their current work—that it was based on the CSB’s job descriptions for HRM, HRD, and management. The job descriptions were new to the participants and were written based on international standards, which were significantly different from the old job descriptions in the MOH. None of the participants had experience with the things being taught; the training was designed to prepare them for the future when the new job descriptions were finally approved.

Training at the MOH: The National MOH Strategy (2018-2022) states,

“The Ministry attaches great importance to the training of the administrative and technical cadres and raising their efficiency through short-, medium-, and long-term plans and procurement of services, residency programs, continuing education, Mission and course assignments, and internal and external conferences according to the needs of the Ministry. In some of these areas, the Ministry receives support through HRH2030. The Ministry is looking forward to expanding the internal and external scholarships and working on its sustainability to meet the shortage of specializations of employees and non-employees in the Ministry, and in particular the scholarship to workers in remote areas to commit to return to their areas and their service.”9

A key official at MOH said training had become very important at MOH, “as evidenced by the requirement that every new practitioner be trained for two months in a hospital, including those coming from being trained abroad, and that administrative promotions be conditional upon completion of a number of training hours.” This official also said training capacity at MOH needed to be developed.

Training plans at MOH and in the directorates, where they exist, are not backed by financial resources. Directorate-level respondents said training events tended to be hurriedly organized at the end of the year when there is a balance of unused funds that would otherwise need to be returned to the Ministry of Finance. Several respondents also said training in MOH focuses on clinical staff, through the HRD department, while training of other technical and administrative staff is addressed on an ad hoc basis. The MOH’s training department coordinates training events when the Ministry is using its own resources, but it is largely dependent on support from international assistance projects.

• Directorate-level plans. There are no formal training plans at the governorate or facility level. Plans may exist for clinical staff, but there are no resources to implement them unless outside sponsorship is found, especially when HRH2030 would not support training. FGD participants from the Ramtha health directorate said they were able to mobilize approximately 100 dinars ($140) from the Jordan Medical Association to conduct general orientation sessions.10

9 MOH Strategy 2018-2022, translated from the original in Arabic. 10 Also cited in HRH2030 Quarter 2 fiscal year 2018 report.


• Hospital training plans. Hospitals generally have training capabilities and plans. Their clinical staff has opportunities for training, but administrative staff seldom does. KIIs and FGDs concur that nursing departments in particular are well-organized and provide training to their staff.

To what extent are staff using their newly acquired skills?

Information in this section is based on views expressed by training recipients and key staff working with them, such as staff they supervise and their own supervisors. Seven FGDs were held with personnel who had received training from HRH2030.

HRM, HRD, and HML training participants seem to be using their new skills, but only to a limited degree. According to FGDs and the quantitative survey, they are not supported in using those skills, and their ability to apply them is limited by lack of time and uncertainty about how to apply the information from the training in practice. As one FGD participant stated, “There were some health directorates which used the forms and templates for training needs assessments and developing training plans that were shared with them during the training. This is because the workload at their directorates is less compared to other directorates.”

The FGDs with staff whose supervisors attended HRH training indicated that the supervisors’ behavior and management had improved. There are signs of leadership, cooperation, and respect for time, which go beyond supervision. Furthermore, there is a difference in the HML trainee’s behavior in areas such as file management, sharing information, day-to-day management, being more responsible, showing flexibility, having more meetings with agendas and minutes, and commitment to regular attendance and taking minutes. The evaluation team heard the following from FGD participants:

• “Some managers are now using performance management approaches they did not use before.”

• “I am more organized; I know how to communicate with my staff better.”

• “The training helped us customize our communication according to the individual.”

FGDs and the survey indicated that some participants in the HML training were changing their management behavior, and that people supervised by managers with HML training were receiving more performance management, such as better communication, team meetings, and appraisals. FGD participants confirmed that in some instances their managers were applying skills they acquired in the Activity’s HML course to change the way work was organized, such as the frequency and quality of staff meetings, and staff consultation for decision-making. One FGD participant said, “I learned how to do annual appraisals and how to give feedback, which was not done before.” Another FGD participant stated that “As a result of training our supervisor received, our center has a newly developed vision and mission statement, we share more. We have more meetings with agendas, minutes, and commitment to regular attendance. We have a form for minutes.”

Around 41.5 percent of survey respondents who were supervised by managers with HML training said they had regular meetings with their management, compared to 25.8 percent of people supervised by managers that did not attend the HML training. Furthermore, the survey indicated that communication between the staff and their management improved among those who attended HML training.

All (100 percent) of survey respondents trained by HRH2030 felt that their training had been useful. This is in stark contrast to criticism of the training shared in KIIs and FGDs. It is possible that survey respondents felt that the training was useful and, simultaneously, that it was also flawed.

Trainees have the ability to apply what they learn, but this is not the norm. A common theme of anecdotal evidence in FGDs and KIIs was that those who applied new learning were motivated and committed to their jobs prior to training. The ability to apply training was also strongly linked to the


quality of the training and its practicality. Some participants felt that some parts of the training were fairly theoretical, and they did not know how to apply it. An HML participant commented, “The model (the McKinsey framework) is just dreams on paper. It cannot be applied in our work.”

Similarly, the findings showed that administrative staff trained by HRH2030 were also rarely able to use their new knowledge and skills. In some cases, their supervisors would not let them apply their training; in others, the training itself wasn’t practical enough. Some motivated individuals indicated they were applying skills and practices they learned in training but said there was not a culture of support around them to do so and their efforts had little chance of being replicated and followed by others. Sometimes, the barriers were simply physical. One HML participant said, “I learned about organizing meetings and wanted to apply it, but there is no room in the facility for a meeting of 27 people.”

The absence of a culture of support manifests itself most directly through the fact that innovators and improved performers receive no reward from the system. They have no better chance for promotion than staff who perform at a much lower level. The quantitative survey showed that only 18 percent of those who were trained and 11 percent of the others felt that MOH recognized good job performance. A somewhat larger proportion of respondents—58 percent of those trained by HRH2030 and 45 percent of others—said they were getting meaningful recognition for doing good work.

The prevailing view is that there the MOH does not have an overall culture of training or capacity building for non-clinical personnel, and the only training opportunities are those brought in through international projects, such as HRH2030. Only 16 percent of the staff surveyed indicated they were satisfied with the job-related training MOH offers, compared to 52 percent satisfaction among those surveyed who participated in HRH2030 training.

Analysis: Why or why not?

A “critical mass” of trained staff has not been created. The Activity reached all directorates with its training in HRM, HRD, and HML, but not a substantial percentage of staff. Training capacity is not yet housed in the MOH or a local institution. The Activity seems to have started training trainers in Year 1 for the supportive supervision course, discontinued training-of-trainers in Years 2 and 3, and then planned for Year 4. Training-of-trainers was initiated along with the HRM/HRD training for the hospitals in Year 3.

Some participants felt that supervisors were reluctant to let their staff go for training. “The directors always give priority to getting the daily work done,” one respondent service provider explained, “so they can rarely accept to let you leave for training.” Some supervisors said they did not know about the training ahead of time and did not get feedback about it afterwards. “I don’t know,” one supervisor said, “I never get any feedback, [staff] come to complain about one thing or another or want to tell me how hard they have been working on the training.” Other supervisors expressed suspicion of training programs in general, suggesting that staff members use them only as a break from work. One official suggested receiving a training attendance report because he believed people used training as an excuse for vacation.

In many cases, resources were not available for trained participants to use new skills. For example, the HRH2030 training modules included skills for how to make training plans; however, when such plans did exist, they could not be implemented due to lack of funds for transportation and meeting costs. As mentioned above, FGD participants from the Ramtha health directorate said they received approximately 100 dinars ($140) from the Jordan Medical Association to conduct general orientation sessions. HRH2030 told the directorate it would not support this training because it was not part of the Activity’s structure.



To what extent are capacity, KM, transparency, and accountability in planning, managing, and retaining Jordan’s health workforce increasing? Why or why not?

Planning: There is no evidence of planning for the MOH health workforce other than what is in the Ministry’s strategy document, which expresses overall staffing targets without a clear plan to achieve them. Key informants indicated there was no succession planning and, other than links with teaching hospitals, there was no evidence of MOH linkages with institutions responsible for producing new cadres.

Using capacity, KM, transparency, and accountability in planning: As discussed above, planning is minimal. However, strategies based on assessments and some level of stakeholder consultation and participation (e.g., the National HRH Strategy) support the conclusion that MOH capacity in planning has increased, its KM has improved (evidenced by assessments and dissemination of their results), and some transparency now exists (evidenced by stakeholder involvement). It remains unknown whether there is ownership and long-term sustainability for such capacity given the high level and intensity of HRH2030 involvement in producing the assessments, developing the strategies, and driving the planning process.

Managing: Signs of improvement include the number of policies and procedures developed; HRMS improvements; interventions to improve employee orientation; newly acquired employee skills; job description development and updates; efforts to determine staffing needs and improve distribution of personnel; efforts to improve women’s participation in management; and improvements in leadership through training. Respondents noted other positive developments, such as the relicensure bylaw and continuing professional development (CPD). Based on feedback from the FGDs and KIIs, these improvements have increased capacity in management and are indicators of that capacity.

However, FGDs and KIIs also frequently referred to the continuation of old hiring practices, improper selection of training participants, lack of transparency in employee appraisals, and lack of fairness in assessing employee performance and determining promotions and incentives. One key stakeholder said, “There is no change on the recruitment process. They are making efforts but there is no change yet,” while an FGD participant noted, “Appointments are done based on the vision and desire of the minister or someone even higher.” There were consistent complaints across the two directorates visited about the central level making uninformed decisions about staffing at the directorate and facility levels due to outdated information. Other complaints concerned staffing decisions being made without consulting immediate supervisors. A FGD participant said, “I never get data from the MOH that is updated or timely.” A senior official at the MOH mentioned in a KII that “accuracy of data is our biggest challenge.”

KM: There have been some improvements in KM, including the availability of data from assessments, studies, and surveys (e.g., WISN), the upgrading of the national health observatory and the HRMS, and efforts in information dissemination, including through social media. Data quality, however, remains an issue, with key people involved in the process citing missing data and delays in updating data due to bureaucracy at MOH. Data entry takes place at the health facilities and is provided on paper to MOH to be handled by the HRMS team. It then goes through different bureaucratic channels, with no guarantee that it will reach the data entry team at the right time, if at all. “Data on HR is not precise,” a service provider said. “For any information on any employee, you have to go back to the paper files and not to the computer.”

The scope of dissemination efforts and the absence of a communications strategy and plan are signs of a weak KM component. In addition, KM is not being applied in decision-making. Generally, managers have a culture of not sharing information, either deliberately to protect personal power or because they do not have the opportunity or capacity to share.


A few FGD participants asserted that information, although not easily available, is obtainable “if you really wanted it.” Most of this group of women said they did not know their rights, but one said she could “actually get enough information from the official website of the MOH and the CSBs [but] if you really want to know something you need to work for it.”

There have been some improvements in data availability. For example, annual reports are available on the MOH website. New and updated tools exist, including WISN, the updated HRMS, competency-based job descriptions, and the relicensure bylaw, but these are not yet in widespread use, and WISN data in particular are not yet reliable enough to be used for staffing decisions. Furthermore, research and assessments are now publicly available to assist in decision-making, such as a mixed-methods study on motivation and retention of health workers in MOH facilities in four governorates and a study on barriers to and enablers of women’s leadership in Jordan’s health sector.

Transparency and accountability in management: FGDs and KIIs indicated an absence of transparency and accountability. Management decisions are not only made without consultation but often at the wrong level, usually much higher than needed. Decisions are made according to inappropriate criteria, such as personal interests, wasta, gender discrimination, and favoritism. Only 21 percent of service providers who responded to the survey felt that recruitment was transparent and fair, and only 18.5 percent felt they had the same chance of being promoted as other staff.

However, there is anecdotal evidence that some managers at health facilities are applying newly acquired skills on communication and staff relations. These managers are having more consultations through staff meetings, and are being transparent in the decisions they make, which are nevertheless limited to their sphere of influence and areas where higher levels allow them some latitude to operate. Overall, there are no indications or perceptions of systemic pressure to support accountability or transparency, and no visible HRH2030 interventions to develop or monitor these behaviors.

Retention: Fewer than half of the service providers surveyed saw themselves working at MOH in five years’ time. KIIs indicated that nothing was being planned or implemented to reduce migration or address the effects of losses due to retirement. Informants also felt that nothing was being done to improve employees’ motivation or job satisfaction. According to one supervisor, “The ability of the system to retain workers is weak; anyone who has an opportunity to leave will do so as soon as they can. There is pressure on employees to work on more tasks without motivation.”

Although there was some positive feedback about availability of breastfeeding hours and flexible hours in facilities and directorates, on the whole, the staff in the directorates cited lack of improvements in their working conditions. Lack of access to computers was frequently mentioned. Women perceived their work environment as unsupportive in allowing mothers to succeed at their jobs.

Increased use of capacity, KM, transparency, and accountability in MOH retention of its workforce: Retention strategies or plans are not evident, and MOH is not using management approaches or data to affect retention. To a great degree, retention depends on compensation, motivation, incentives, and financial resources, none of which MOH plans to modify. The Ministry’s leadership asserts that the only retention issue is for specialist doctors, who are being lost to the private sector or to migration. They argue that there are enough non-specialist professionals and support staff, and the application of the mandatory retirement law is not an issue.


To what extent is the Jordan HRH2030 Activity influencing women’s management and leadership in the health workforce?

There are two ways in which HRH2030 has the potential to influence women’s management and leadership in the health workforce: supporting culture change for data-based promotion practices at


MOH and implementing specific activities to increase opportunities for women. The first is a long-term approach, as culture change takes time; the second approach has more potential for short-term impact, as it supports individual women to overcome cultural and institutional barriers.

Data collection for this evaluation took place in September 2018, when the Activity had been in place for three years and actively working with MOH for two years. HRH2030 launched its first piece of research on barriers to women’s leadership in June 2018 and established a forum for women in the health sector in mid-September 2018. At the time of the evaluation, HRH2030 had not had any influence on women’s management and leadership in the health workforce. However, this appears to be due to the Activity’s timeline rather than the quality or appropriateness of its interventions.

It is unrealistic to expect culture change that leads to additional support for women in leadership positions in two years. This is especially true when the goal of that culture change is widespread HRH change and not solely the advancement of women.

The evaluation team did find evidence of such change. There was wide consensus among KII and FGD participants that the hiring and promotion system was unfair to both men and women. They felt that favoritism, nepotism, and connections played a significant role in the advancement of less qualified/deserving people, regardless of gender. They also identified a lack of transparency in announcing open positions, opportunities for promotion, or calling for applications. There seems to be no obvious in-house mechanism for appeals against unfair or missed promotions. Some said the MOH avoids announcing open positions because it would result in thousands of personal interventions to influence the decision, causing more problems than just making a non-transparent decision. Only 21 percent of service providers surveyed felt that recruitment was transparent and fair, and only 18.5 percent felt that all staff had an equal chance of being promoted. Conversely, female physicians at the directorate level who participated in FGDs were positive about the fairness in the system and felt that it allowed them to have access to positions of decision-making.

The evaluation did find political will at MOH to address barriers and advocate for better access to leadership. It is understood that there is very high-level support—from above the Ministry level—to increase the number of women in leadership roles. As such, the MOH is seeking to promote women into visible roles.

However, the Ministry is not ready to make changes that would make the roles more welcoming to women. The consensus among both KII and FGD participants was that women who accept leadership roles face unusual hardships and opposition from their male peers, and are expected to handle this opposition on their own. For example, the evaluation team heard the following comments from a leader in a women’s FGD: “We have two or three examples of women who were appointed by MOH as directors at the central level and faced a strong and continuous opposition within their staff and the community around until they quit. I would not want to be in such a position and will not accept that.” and “Recent appointments were for women in leadership posts and there were some negative remarks made on Facebook against them.”

The appointment in late September of a woman as hospital director was a much talked-about event in the Jordanian media and at MOH. Hailed as a breakthrough, it was an important step in breaking down gender stereotypes about which jobs women can or cannot do. In the women’s FGDs, several conversations centered around whether any of the female supervisors would be willing to take the job of hospital director. All agreed when one participant stated that “it is not a job for a woman as it involves impossible hours, dealing with community anger and violence over health care access issues, dealing with crimes and emergencies.”

The evaluation also found that HRH2030’s research did not have an effect on barriers to women’s leadership in health or the Women’s Forum. However, as noted above, the research was released in


June 2018, just three months before evaluation data collection. The Women’s Forum began during the data collection period, and it is not plausible to expect that these interventions could show results in such a short period.

According to key informants, the HRH2030 research on barriers to women’s leadership and career advancement provided respected and reliable details about the challenges women face in health care, adding that the results were accurate and carefully researched. Furthermore, women FGD participants confirmed the conclusions of the research. However, the research does not appear to have been widely disseminated. While many KII participants were aware that the research had taken place, they were unfamiliar with its contents. Most female FGD participants were unaware of the research or its release.

The Women’s Forum and the Women in Leadership Network were popular ideas among key informants, including those who worked directly on women’s issues. There was consensus that this approach had worked to support women in other sectors and had potential for success in the health sector. In addition, the evaluation team observed enthusiasm and excitement when the Network was mentioned at one of the women’s FGDs outside Amman. Through the remainder of the Activity, regular meetings and support to increase membership will put the Network on the path to sustainability.


What are the next steps for institutionalizing the new law that requires continuous professional development for Jordan’s health workforce?

This question relates to the bylaw on relicensing of health professionals, “Bylaw #46 year 2018,” for the “renewal of licensing for the health professional.” This bylaw mandated CPD as a prerequisite for relicensing and established a National Committee for CPD with the purpose of “raising the quality and safety of health services in Jordan by providing guidance to the establishment and sustainability of a national continuing professional development system that promotes competency development for all licensed health professionals in Jordan.”

Due to the nature of this question, the discussion below combines true findings with analysis conducted by the evaluation team. The terms “relicensing” and “CPD” have been used in the same context and interchangeably in our discussions in KIIs and FGDs.

Bylaw Development and Approval Process: A series of steps involving different levels of government was necessary to develop and approve the bylaw. First, the

1) A committee was formed by the Minister of Health, who chairs the HHC.

2) The committee submitted the draft to the Minister.

3) The Minister sent the draft to the MOH legal advisor for final review and edits.

4) The Ministry submitted the draft bylaw to the Prime Ministry.

5) The Prime Ministry submitted it to the Opinion and Legislation Bureau.

6) The Opinion and Legislation Bureau invited the committee members, including the MOH legal advisor, to discuss required changes.

7) After agreeing on the changes, the Bureau submitted the revised bylaw to a ministerial committee at the Prime Ministry for approval and dissemination in the official gazette. (The King signs many of the regulations.)

8) The Opinion and Legislation Bureau sent the instruction back to the MOH to discuss with the associations.


9) Although the associations are represented in the National CPD Committee, which had approved the instruction, some had additional comments and not all agreed to the instructions as sent for final approval.

10) At the time of the evaluation, the HHC was still trying to obtain consensus among the councils and associations.

Qualitative data collection found that some stakeholders and health professionals were aware of the new CPD requirement for relicensing, but very little clarity about its objectives and implications, the modalities of its implementation, or the sources of professional education. As the bylaw implementation instructions are currently in final draft form and have not been approved, it is understandable that they would not be common knowledge. However, it is unusual that there is so little awareness within the health professions, and that the stakeholders express so much opposition and skepticism about the justifications for and potential benefits of relicensing. FGD participants had only a minimal sense of what the bylaw required, and generally did not express an understanding of its purpose. Key informants had a greater level of awareness, implying that leadership had a better understanding than the facility- and directorate-level professionals who participated in focus groups.

Stakeholder views on CPD range from considering the bylaw as an important step that will drive cultural change around CPD to seeing it as a dead end that will never be successfully implemented. Some FGD participants expressed concerns about commercial interests taking over and the risks of fraud or sale of CPD/continuing medical education (CME) credits. An alternate sentiment was that continuing education was “very important; we normally are lazy or want something the easy way, so unless we are obligated to do something, we won’t do it.”

Others thought the bylaw risks exacerbating inequities and could actually harm access to health care by decreasing its availability in rural areas. For example, one key informant was concerned that “the new law may lead to staff refusing to go to distant health facilities.” Some key informants also expressed concern that the bylaw had been put into place without sufficient structure to support it.

Stakeholders who had participated in HRH2030 training had a better impression of the new bylaw than those who had not. According to the quantitative survey, 51 percent of health staff who received training were aware of the bylaw for renewal of health professional licenses and 57 percent viewed it as an opportunity to improve medical care. (Note: more people approved of the bylaw than were aware of it; this may indicate general support for relicensing rather than the bylaw itself.) Of those who did not attend training, 31 percent were aware of the bylaw and 22 percent approved of it.

Why is there opposition to relicensing/CPD?

Until now, CPD had been the privilege of those who could afford it, namely physicians and, to a lesser extent, dentists, nurses, and pharmacists. As a result of the new bylaw, all health professionals will be required to have a certain number of hours of CPD every five years. For a majority of the 160,000 targeted health professionals, their introduction to the concept and practice of CPD will not be voluntary—it will be a requirement for relicensing. Minimal awareness of the new bylaw suggests that MOH and HHC have devoted very few resources to campaign or advocate for it.

HRH2030’s role in support of HHC and MOH has included working with stakeholders, including associations representing the health professions and the official councils that regulate them. However, it appears that these bodies have not made significant efforts to educate their constituencies and garner support for the new bylaw, which they helped develop. According to one key informant, “The prevalent view of how things work in Jordan is that laws are promulgated and then the law-abiding nature of the population will make things work out just fine.”


Where is support for the new bylaw coming from?

Evaluation interviews found support for relicensing and CPD among various health professionals, primarily in professional associations and councils, the accreditation council, and other potential providers of CME. Based on information from the Private Hospital Association, private hospitals are thinking about some form of collaboration for developing or sharing their capacities in CPD. Many of these hospitals acquired some capacity for CPD as part of their upgrade to gain accreditation; they expect to have a role in providing CPD.

There is no clear consensus among stakeholders about the next steps needed to implement the bylaw. Some stakeholders object to the current version of the implementation procedures, but the new instructions are with the minister of health for approval/signature. They could be signed without further consultations.

HRH2030 was a key player in the process leading to the promulgation of the bylaw and drafted both the implementation instructions and the initial text of the bylaw. HRH2030 leadership said the Activity prepared the draft through consultations and submitted it to the Minister’s office; when it reemerged, it was a significantly altered document. It is not clear who made these substantive changes or which, if any, stakeholders were involved, but the evaluation team has interviewed key stakeholders who appear to have been left out of that process.

It is not known if all those who still have concerns will have sufficient access to the minister to make their voices heard and obtain an additional round of consultations. It would not be unusual for a decision to be made at the higher levels without having gained full consensus from the stakeholders.

Next steps

Institutionalizing the new bylaw will require a wide range of next steps. These could be supported through assistance to a single entity, such as a government institution or a USAID project, that would focus on driving these processes, or through support that is micro-targeted across Jordan’s health sector. (This will be further elaborated in the findings under Evaluation Questions 6 and 7.) The next steps include the following:

Dissemination: The bylaw and its implementation instructions need to be widely promulgated, beyond the usual approach of merely publishing in the government gazette. To create a culture of CPD that will genuinely impact care, a “critical mass” of health professionals need to understand and support CPD and the governance around it. Dissemination efforts could include social media, mass media campaigns, and facility-based education.

Development of CPD capacity for remote areas: Although private hospitals and professional associations and councils are planning to provide CPD, it will most likely be targeted at providers in the major cities, which have the resources to pay for CPD. This means providers in remote areas or those with low salaries might not be able to access the CPD they need to remain in practice. Therefore, a government entity needs to develop CPD opportunities for providers who cannot travel to major cities or pay for expensive programs. This could be Web-based education, teleconferences, or travelling training programs that visit each governorate. MOH is the entity best-positioned to provide this type of CPD, in collaboration with other public universities/institutions and health professional associations and councils.

Monitoring and registration: MOH, as the licensor of all health professionals, needs to have the capacity, with its affiliated health professional councils, to track which health care providers have met their CPD requirements. This includes a system for recognizing accredited CPD programs and accepting provider-submitted evidence of learning.


Coordination: Major providers of CPD are likely to include, as per the draft instructions, the MOH, the Royal Medical Services, the Jordan Medical Council, the Jordan Nursing Council, health professional associations, and public and private teaching hospitals. Coordination among these entities, especially for online training, through a recognized institutional arrangement will improve the overall quality of CPD.


How can USAID best sustain the achievements and reforms accomplished under HRH2030 in Jordan?

HRH2030 has a broad mandate, and as a result, it supports a wide range of interventions intended to strengthen HRH in Jordan. Under this question, the evaluation team attempted to identify the specific topics and processes among those interventions that are likely to have the most impact if supported beyond the life of the Activity. (Question 7 addresses potential procurement and design approaches that USAID could take to support these topics.)

Support the relicensure bylaw: As discussed under Evaluation Question 5, the relicensure bylaw has the potential for substantial positive impact on the quality of health care in Jordan. On the other hand, it could amount to nothing, or even cause harm. Also, stakeholders noted the issue of who will “own” the bylaw, as no single entity is responsible for its implementation. USAID could support a stand-alone activity that serves as that entity. It could advocate for the bylaw and work to ensure that appropriate implementation structures are put into place for its enforcement. This activity could also provide technical support in dissemination, capacity development, monitoring and registration, and coordination.

1) Engage the Civil Service Bureau: Because health care providers in the MOH sub-sector are government employees, there is a limit to how much MOH can affect HRH. The major HR functions for government employees sit with the CSB, which is responsible for setting regulations for hiring, compensating, promoting, retirement, and monitoring performance of all civil servants, including HRH working in MOH.

Despite expressed MOH support for the WISN tool in health workforce planning, it is doubtful that the CSB will formally use it to allocate human resources to MOH by the CSB. USAID could provide support to the CSB to revise the civil service bylaws, including the articles related to the MOH HRH allocation, promotions, and retirement. Support could be extended to expand the MOH-CSB harmonized HRMS down to the health directorate level.

The current financial incentives for health professionals working at MOH are not linked to performance or serving in remote areas. A financial and non-financial incentive system, approved by CSB and linked to performance and serving in remote areas, could enhance productivity and the responsiveness of the health workforce while improving access to health care in underserved areas.

Other topics for collaboration include revising job descriptions, changing promotion criteria for health care providers to include CPD and quality of work, and addressing causes of job turnover at MOH. USAID could engage CSB on HRH topics, either through an implementing partner activity or CPD a government-to-government mechanism.

2) Continue support to the Women in Leadership Network: Although it is a widely supported activity that stakeholders believe has a lot of potential, at the end of HRH2030, the Network (forum) will be only two years old. It is unlikely that it will be sustainable at that time. USAID could continue support to the Network until it grows into a self-sustaining entity.

3) Institutionalize HML training: USAID could support the institutionalization of HRH2030 HML training, at MOH or the Ministry of Public Sector Development/National Institute for Training and other accredited training providers. This would build capacity for health management within the


MOH system and more broadly throughout the health sector. Although feedback on the HML training was mixed, the content of the training is relevant to Jordan’s health system, and a fully institutionalized training approach could maintain and evolve the curriculum to optimally support leadership development and HRH.


What does the evaluation team recommend regarding any potential follow-on HRH activity? If a follow-on is necessary, should it be similar to HRH2030, or be combined with another existing USAID/Jordan activity, or take the form of direct support to the government of Jordan through the utilization of partner government systems?

1) USAID should consider reducing the number of awards but implement larger activities: MOH and USAID partners indicates a concern about “too many projects.” MOH counterparts said they spent a substantial amount of time responding to requests from USAID activities. One HRH2030 counterpart at MOH estimated that he devoted 60 percent of his time to HRH2030. While this is probably an exaggeration, it indicates that this person perceived HRH2030 as a burden on his time.

Partners have also indicated that they are devoting substantial time to handling or preventing issues that stem from coordinating the four health sector projects in the country (the Health Service Delivery Activity, the Jordan Communications, Advocacy, and Policy Activity, the Jordan Health Finance and Governance Activity, and HRH2030). A review of reports from these USAID activities in Jordan revealed that some programs or areas were on more than one activity’s work plan (e.g., HRH2030 developed the community feedback mechanism, which was implemented by the Health Service Delivery Activity), and USAID decided to withdraw it from one Activity and allocate it to the other within its portfolio. Still, these activities have potential for coordination and synergies. For example, HRH2030’s work such as in governance, training, communication, policy development, and decentralization are also important components of the other USAID Activities.

2) USAID should consider combining service delivery and infrastructure projects with interventions that pose more of a challenge to stakeholders (e.g., HRH): This would allow the more stakeholder-challenging interventions to leverage the relationships and good will created by infrastructure activities. The KIIs, FGDs, and the quantitative survey all found that staff at the directorate level and the MOH central level were concerned with working conditions, availability of computer equipment, funding for training plans, and building training capacity.

Because HRH2030 is not designed to address any of these needs, the same entities it relies on to implement its HR improvement programs could see the Activity as being unhelpful. This is especially challenging because counterparts do not regard HRH reforms as being critical to Jordan. USAID could “sweeten” such challenging interventions by providing equipment, services, and resources to run activities that MOH is unable to support.

3) USAID should consider a multisectoral approach to HRH that involves multiple ministries, including the Ministry of Finance, the Ministry of Higher Education and the Ministry of Public Sector Development. HRH in Jordan is under the purview of several public sector entities and involves a multitude of stakeholders, as enumerated in the HHC HRH report of 2015. On numerous occasions, FGD and KII respondents said the MOH was largely dependent upon the CSB for many of the core areas within its HRM. They also mentioned the important role of the Ministry of Finance. In light of such feedback, USAID should consider working with.

USAID could also consider working with some of the many other entities involved in Jordan’s health sector. These include Parliament; the Council of Ministers (Cabinet); the Higher Education Council;


the Ministry of High Education and Scientific Research; the High Education Accreditation Commission; HHC; health professional associations; the CSB; professional councils (e.g., medical and nursing); teaching hospitals; the Royal Medical Services; private hospitals; the United Nations Relief and Works Agency for Palestine Refugees in the Near East; the philanthropy health sector; and the Consumer Protection Society.

The evaluation team understands that a strategic decision was made to limit the scope of HRH2030 to the MOH health subsector. Although this decision was no doubt essential at the time, it did limit the Activity’s potential impact. Any comprehensive effort to significantly improve the HRH sector will need to extend beyond MOH.

4) USAID should consider adding a grant-making function to any new HRH activity: This would allow the activity to flexibly respond to unanticipated needs and provide direct support to emerging partners in the HRH space.

5) USAID should consider an activity that supports decentralization: MOH is not currently prepared for decentralization, despite increasing interest at the highest levels of the Jordanian government. The MOH’s Decentralization Directorate is new, and the head of the directorate is inexperienced. No stakeholder within the MOH system appears to have a clear sense of the risks and benefits of decentralization. As WHO states in its 2017 governance assessment:

“The HHC is not present at decentralized level while the MOH is organized by directorate and can easily mobilize these peripheral entities in the implementation of decentralization. However, the exact delegation of authority to the directorates is yet to be determined as well as the distribution of key functions (planning, policy, regulation, etc.) between the elected bodies at provincial level and the health directorates.”

There are many choices to be made about which responsibilities and which authorities to delegate during decentralization. The financial implications alone are extremely complex. A carefully designed USAID activity could provide technical assistance to MOH during the decentralization process.

6) Finally, USAID should consider working with MOH and governorate level/local leaders to improve management and leadership capacity at the directorate level: This would support success in eventual decentralization and help support the even distribution of non-medical capacity within the MOH system.

CROSSCUTTING FINDINGS

Decentralization

A situation analysis from USAID’s Health, Finance, and Governance Activity’s, Decentralization in the Public Health Care in Jordan,11 documents the country’s decentralization efforts and provides evidence that cautions against assuming that decentralization will guarantee improved health care. It points to several areas where Jordan is not prepared for decentralization, including a fragmented health care system, lack of accountability, and lack of citizen input into governance. This is compounded by the poor level of public awareness about decentralization and its likely impacts.

KIIs with health professionals revealed little awareness and some fear of the potential impacts of decentralization. There is deep uncertainty about how decentralization will impact the health sector. Some respondents echoed the need for incremental levels of autonomy to be given to hospitals beyond their current 200 dinar ($280) spending authority, and one respondent pointed to “the abnormality of a

11 Decentralization in the Public Health Care of Jordan, USAID Health, Finance, and Governance Activity, 2017.


system where a major hospital did not have the authority to authorize repair for a burst toilet and had to wait for a decision from the central MOH level.” As stated above, staff at the MOH’s new Decentralization Directorate while skilled, are inexperienced.

Communications

The evaluation team found that HRH2030’s communication efforts were missing opportunities to increase awareness of HRH and of the Activity itself. The lack of awareness about the National HRH Strategy within MOH and the health workforce in the private and civil society sectors indicates limited communications and insufficient participation and representation. A strong program of systematic dissemination should be in place to support all new regulations and procedures; this can be developed in with other USAID projects, such as the Jordan Communications, Advocacy, and Policy Activity.

Advocacy for HRH

Overall, the evaluation did not find evidence of a cultural change at MOH in support of better HRH. The level of national ownership of HRH2030 interventions seemed low. There was a perception among stakeholders that the Activity simply drafted and developed materials then asked them to sign off on them, rather than engaging in a true participatory process.

The National HRH Strategy (2018-2022) and the National MOH Strategy (2018-2022) were developed in parallel during overlapping time periods with the support of two outside experts, a national expert for MOH with WHO support and an international expert for HHC with HRH2030 support. The experts talked to each other and visited Jordan at various times, but the two strategies are not necessarily synchronized or aligned. It is outside the scope of this evaluation to identify the areas of discrepancy or gaps; however, a preliminary review showed that the strategies have differing degrees of elaboration of details and development of measurable targets.

The fact that an international (i.e., non-Jordanian) expert played a key role in developing the National HRH Strategy affected stakeholders’ perceptions of it. Key national stakeholders said they were not involved on an equal footing in its development. Instead, they reported that the consultant simply briefed them about drafts before he sent them to HRH2030 for review and approval. Several respondents were skeptical about strategies in general, feeling they were seldom implemented because they were not backed by operational plans and resources, and remained just documents on shelves.


V. CONCLUSIONS AND RECOMMENDATIONS The conclusions and recommendation provided below are based on the evaluation team’s analysis of the findings. They represent the combination of the team’s expertise and the data gathered during the evaluation process.

CONCLUSIONS

1. Human Resources Practices

The HRH2030 Activity is concerned with improving HR practices. Long-term improvement will require scaling up the best practices it has initiated. However, scaling up is not mentioned anywhere in the Activity’s documents, and scale-up is not possible using approaches that rely solely on HRH2030’s own resources and staff without engaging Jordan’s capacity and resources.

There are policies and tools available at MOH for motivated managers to utilize. Some managers at the directorate level are now using performance management approaches they did not use before and applying some of the tools and skills acquired during their training, but this is the result of personal initiative rather than a cultural shift. The perception within MOH and from key informants familiar with its systems is that there is little change in culture and practice regarding HRH, and not enough ownership or use of the new tools.

The success of the WISN method is threatened by issues with data quality and a dearth of staff who understand the Activity beyond data collection. Opposition is anticipated from many quarters to changes in staffing, especially redistribution or reductions as prescribed by WISN results and per the CSB’s current position.

One major challenge is that knowledge about HRH2030 and its interventions is very scarce and poorly disseminated. The absence of a clear communications strategy and advocacy within MOH at central and directorate levels and with stakeholders does not foster transparency, participation, and accountability, and is detrimental to the visibility of MOH achievements and to garnering support for change and new policies. It perpetuates the current perception that little or nothing has changed for the better at MOH.

Improvements cited in HRH2030’s reports cover most of the practices it targeted, such as planning, training and development, HRMS personnel policy and practice, HRM and HRD data, performance management, and training and staff capacity. However, sustainability is in question. There were no visible interventions or changes in motivation systems, career paths, and succession planning. Such changes would have to come from working with the CSB, which HRH2030 has not done beyond HRMS.

2. Training received and in use

There are not enough trained people (in HRM, HRD, and HML) in any of the institutions in the evaluation sample to create a “critical mass” for change at the directorate or facility level. Addressing nationwide training needs using only HRH2030 trainers is not possible and is unlikely to result in the critical mass of trained managers, administrators, and leaders needed to catalyze visible change throughout the system. There is a small number of highly motivated individuals within the MOH system who welcome HRH2030 interventions and tools. These are the people who put training skills into practice, welcome WISN, and engage with the National HRH Strategy. However, there are not many of these people, and they are not connected to each other.

3. Increased use of capacity, knowledge management, transparency, and accountability in the process of planning, managing and retaining Jordan’s health workforce


Overall, there are no indications of a systemic drive toward accountability or transparency. There are virtually no visible interventions to develop or monitor these behaviors, one exception being the use of the MOH website and social media to provide information and data and encourage some degree of stakeholder involvement. Strategies developed based on assessments and stakeholder consultation and participation (e.g., the National HRH Strategy) indicate improved MOH capacity for planning. KM has improved as assessments have taken place and their results shared. No improvements have been observed in workforce motivation or job satisfaction, but some improvements are evident in performance management, including the move toward updated job descriptions, the use of appraisals, and some participation in decision-making at directorate and facility levels.

4. HRH2030 Activity influencing women’s management and leadership in the health workforce

There appears to be strong support from MOH leadership and the government of Jordan to increase the number of women in leadership positions. For this to succeed, this push needs to be part of a gender-transformative leadership approach that aims for a fundamental, lasting change in the structures and cultures of societies and organizations. International assistance, including from Oxfam and WHO, offers training programs, guidelines, and manuals about gender-transformative leadership and policies that may benefit leaders in MOH and in the Women in Leadership Network.

The influence of HRH2030 and the strong leadership of USAID are credited with the establishment of the Network/Forum and may have created expectations that support for this initiative will continue after the Activity ends. There is a level of expectation and excitement among women at both central and directorate levels — that can energize this new initiative. Existing gender programs within USAID and the government of Jordan, such as the Jordanian National Commission for Women, offer opportunities for synergies and coordination.

5. Next steps for institutionalizing the new law that requires continuous professional development for Jordan’s health workforce

Most stakeholders know little or nothing about the relicensure bylaw, from its rationale and justification to the details of its application. There are mixed views on the impact it will have on health professionals, quality of care, and the access to care in remote areas. There is a lack of consensus about the implementation procedures and a need for further consultations to resolve outstanding issues. There is a further need for extensive communication about the new bylaw that should be addressed with the cooperation of professional associations. MOH, the regulatory bodies in charge of registration and monitoring mechanisms, and potential providers of continuing education face a strong challenge in establishing these systems.

RECOMMENDATIONS

General recommendations

1. Develop a project-level sustainability plan to ensure the long-term impact of interventions: At present, each HRH2030 intervention has a sustainability component, but there is no overall sustainability plan. Such a plan would support “big picture thinking” and increase the likelihood of long-term impact.

2. Seek high-level buy-in at MOH in addition to focusing on mid-level officials: The evaluation team consistently found minimal awareness of HRH2030 interventions beyond immediate MOH counterparts. As HRH2030 is a non-clinical activity, it can be difficult to build support for its work, and the lack of awareness compounds the problem. Furthermore, real change at MOH will require support from the highest levels, as change rarely flows upward in Jordan. Gaining top-level support at the Ministry would increase the likelihood of long-term impact and system change.


3. Increase communication and advocacy efforts to disseminate products such as the relicensure bylaw and the National HRH Strategy: Although HRH2030 is not formally responsible for dissemination, at present, there does not appear to be any communication about the bylaw and the National HRH Strategy. HRH2030 should support and advocate for better awareness of these topics.

4. Seek ways to further incorporate health care consumer views into policy: Patients and other consumers have played a minimal role in the discussions around HRH. The Activity should seek ways to better incorporate the views of these important stakeholders in discussions.

5. Prioritize local expertise and institutions whenever possible: Due to the challenges of contracting and deadlines, HRH2030 has not been able to move beyond hiring local consultants into actually contracting local institutions. This is a considerable shortcoming, as working with these institutions would help to build national capacity on HRH topics and support long-term impact.

HRH practices (Evaluation Question 1)

6. Support directorate-level implementation of training plans to scale up the adoption of newly developed HR tools and products: These include orientation for new employees, job descriptions, and personnel appraisals.

7. Assess WISN progress in a stakeholder consultation process that includes the facility level before continuing rollout: WISN currently faces several challenges. Mid-level personnel are uncertain about its value and purpose, the data it produces are currently unreliable, and there is some doubt as to whether WISN data will actually be used even once it is deemed reliable. A consultative process that includes skeptical lower-level personnel could provide useful information as HRH2030 seeks to move WISN forward.

8. Re-examine assistance to the National HRH Observatory: At present, the Observatory exists only on paper—it has no physical or juridical presence. It has very weak technical and organizational capacity and is not a useful source of data. It will require substantial outside assistance for the observatory to be able to serve its intended function. HRH2030 can provide some assistance, but cannot meet the full need. If no other partner provides support, the Activity’s efforts will have minimal impact. It is likely that the objective of reliable health system data can be better pursued through a different approach.

9. HRH2030 staff gave lectures about HRH at Al-Balqa Applied University to raise awareness among future doctors, and these were added to the school’s curriculum. A similar module developed for MOH senior staff could be shared across the Ministry, to complement a deliberate effort to disseminate HRH2030 literature more broadly.

HRH training (Evaluation Question 2)

10. Develop training capacity of MOH at the national, directorate, and hospital levels: All training capacity rests with HRH2030. This does not support long-term sustainability. The Activity is developing packages to support the transition of training to MOH, but it should consider co-developing these with trainers from the Ministry. The handover should begin as soon as possible, while HRH2030 still has time to support MOH for assuming responsibility for training.

11. Consider developing a core team of national and regional trainers to enable MOH to scale up the training nationwide: This would create training capacity owned by MOH, enable scale-up, and empower the Ministry’s central and directorate levels to take charge of fulfilling its training needs.


12. Consider delivering one HML training module per week instead of one per month: This accelerated schedule would complete training earlier and free HRH2030 resources in the next two years to focus on other priority activities.

13. Explore institutionalizing HML training: This can be done within existing academic programs and extension courses in management and leadership at national and private universities and the National Institute for Training (under the Ministry of Public Sector Development). It would also support MOH at the national and directorate levels to sponsor participants.

14. Design training to target organizations and the environment in which their employees work: A transition to training interventions that target all technical and administrative staff in directorates, hospitals, and health clinics could build common understanding and mutual support across disciplines and throughout the organizations.

Capacity, KM, transparency, and accountability (Evaluation Question 3)

15. Support further capacity development through training-of-trainers and dissemination of existing and newly developed HR management tools such as job descriptions, appraisals, and new employee packages.

16. Improve KM at MOH through a sustained effort to disseminate and communicate within the Ministry’s system, with its stakeholders, and with the public at large.

17. Raise awareness and knowledge among MOH staff and stakeholders about existing and new policies, and their rationale and implications.

18. Further develop the use of available tools, such as the MOH and HHC websites and social media, to improve transparency and accountability.

Women’s access to management and leadership (Evaluation Question 4)

19. Further disseminate the results of research on women’s access to leadership nationally and at the directorate level. Promote thinking, strategizing, and planning for interventions based on the research results.

20. Involve women from the directorate level in the Women in Leadership Network. This could be done through electronic communication and by establishing local networks in the directorates, with representation at the national level.

21. Look at other women’s networks as a model for the Women in Leadership Network and establish linkages with gender programs supported by USAID and other development partners, as well as within the government of Jordan, such as the National Women’s Committee.

22. Familiarize senior MOH officials and Women in Leadership Network leaders with gender-transformative leadership approaches and training programs, such as those supported by Oxfam, WHO, and Women in Global Health.

Continuing professional development (Evaluation Question 5)

23. Raise awareness among providers of the CPD relicensure bylaw through mass media and social media outreach and work with professional associations.

24. Facilitate and support coordination and collaboration among providers of CPD to maintain quality and increase access.


Crosscutting issues

Decentralization

25. Improve the level of awareness among MOH staff at the central and directorate levels about decentralization to improve true understanding of its meaning and scope.

26. Assess the needs of the MOH’s new Decentralization Directorate and support its capacity development.

Communication

27. Widely distribute HRH2030 and MOH public information materials to all Activity stakeholders, including MOH leadership. Consider developing targeted/tailored information for each of these stakeholders.

Advocacy for HRH

28. Support alignment of the National HRH Strategy implementation plan with the MOH strategy action plan.

29. Develop a coherent approach with and by the HHC, involving the MOH and other stakeholders, to properly disseminate knowledge about the National HRH Strategy and develop commitment to its effective implementation.


ANNEX I. SCOPE OF WORK Assignment #: 590 [assigned by GH Pro]

Global Health Program Cycle Improvement Project (GH Pro)

Contract No. AID-OAA-C-14-00067

EVALUATION OR ANALYTIC ACTIVITY STATEMENT OF WORK (SOW) Date of Submission: 5-21-18

Last update: 8/01/18

I. Title: USAID/Jordan Human Resources for Health in 2030 (HRH2030) Midterm Activity Evaluation

II. Requester / Client ☑ USAID Country or Regional Mission Mission/Division: Jordan / Population and Family Health Office (PFH)

III. Funding Account Source(s): (Click on box(es) to indicate source of payment for this

assignment) ☐ 3.1.1 HIV ☐ 3.1.2 TB ☐ 3.1.3 Malaria

☐ 3.1.4 PIOET ☐ 3.1.5 Other public health threats ☑ 3.1.6 MCH

☑ 3.1.7 FP/RH ☐ 3.1.8 WSSH ☐ 3.1.9 Nutrition ☑ 3.2.0 Other (specify): Objective 6: Program Design and Learning

IV. Cost Estimate: (Note: GH Pro will provide a cost estimate based on this SOW)

V. Performance Period

Expected Start Date (on or about): August 6, 2018 Anticipated End Date (on or about): February 22, 2019

VI. Location(s) of Assignment: (Indicate where work will be performed)

Amman, Jordan, with some travel within Jordan, and some work conducted remotely

VII. Type of Analytic Activity (Check the box to indicate the type of analytic activity) EVALUATION:

☑ Performance Evaluation (Check timing of data collection) ☑ Midterm ☐ Endline ☐ Other (specify):

Performance evaluations encompass a broad range of evaluation methods. They often incorporate before–after comparisons but generally lack a rigorously defined counterfactual. Performance evaluations may address descriptive, normative, and/or cause-and-effect questions. They may focus on what a particular project or program has achieved (at any point during or after implementation); how it was implemented; how it was perceived and valued; and other questions that are pertinent to design, management, and operational decision making ☐ Impact Evaluation (Check timing(s) of data collection) ☐ Baseline ☐ Midterm ☐ Endline ☐ Other (specify):

Impact evaluations measure the change in a development outcome that is attributable to a defined intervention. They are based on models of cause and effect and require a credible and rigorously defined counterfactual to control for factors other than the intervention that might account for the observed change. Impact evaluations in which comparisons are made between


beneficiaries that are randomly assigned to either a treatment or a control group provide the strongest evidence of a relationship between the intervention under study and the outcome measured.

OTHER ANALYTIC ACTIVITIES ☐ Assessment

Assessments are designed to examine country and/or sector context to inform project design, or as an informal review of projects.

☐ Costing and/or Economic Analysis Costing and Economic Analysis can identify, measure, value and cost an intervention or program. It can be an assessment or evaluation, with or without a comparative intervention/program.

☐ Other Analytic Activity (Specify)

PEPFAR EVALUATIONS (PEPFAR Evaluation Standards of Practice 2014) Note: If PEPFA-funded, check the box for type of evaluation ☐ Process Evaluation (Check timing of data collection)

☐ Midterm ☐ Endline ☐ Other (specify): Process Evaluation focuses on program or intervention implementation, including, but not limited to access to services, whether services reach the intended population, how services are delivered, client satisfaction and perceptions about needs and services, management practices. In addition, a process evaluation might provide an understanding of cultural, socio-political, legal, and economic context that affect implementation of the program or intervention. For example: Are activities delivered as intended, and are the right participants being reached? (PEPFAR Evaluation Standards of Practice 2014) ☐ Outcome Evaluation Outcome Evaluation determines if and by how much, intervention activities or services achieved their intended outcomes. It focuses on outputs and outcomes (including unintended effects) to judge program effectiveness, but may also assess program process to understand how outcomes are produced. It is possible to use statistical techniques in some instances when control or comparison groups are not available (e.g., for the evaluation of a national program). Example of question asked: To what extent are desired changes occurring due to the program, and who is benefiting? (PEPFAR Evaluation Standards of Practice 2014) ☐ Impact Evaluation (Check timing(s) of data collection)

☐ Baseline ☐ Midterm ☐ Endline ☐ Other (specify): Impact evaluations measure the change in an outcome that is attributable to a defined intervention by comparing actual impact to what would have happened in the absence of the intervention (the counterfactual scenario). IEs are based on models of cause and effect and require a rigorously defined counterfactual to control for factors other than the intervention that might account for the observed change. There are a range of accepted approaches to applying a counterfactual analysis, though IEs in which comparisons are made between beneficiaries that are randomly assigned to either an intervention or a control group provide the strongest evidence of a relationship between the intervention under study and the outcome measured to demonstrate impact.

☐ Economic Evaluation (PEPFAR) Economic Evaluations identifies, measures, values and compares the costs and outcomes of alternative interventions. Economic evaluation is a systematic and transparent framework for assessing efficiency focusing on the economic costs and outcomes of alternative programs or interventions. This framework is based on a comparative analysis of both the costs (resources consumed) and outcomes (health, clinical, economic) of programs or interventions. Main types of economic evaluation are cost-minimization analysis (CMA), cost-effectiveness analysis (CEA), cost-benefit analysis (CBA) and cost-utility analysis (CUA). Example of question asked: What is the cost-effectiveness of this intervention in improving patient outcomes as compared to other treatment models?

VIII. Background

If an evaluation, Project/Program being evaluated: Project/Activity Title: Human Resources for Health in 2030 (HRH2030) Award/Contract Number: AID-OAA-A-15-00046 Award/Contract Dates: January 1, 2016 to August 31, 2020


Project/Activity Funding: $11.6m for Jordan Activities Implementing Organization(s): Chemonics International Inc. Project/Activity AOR/COR: AOR: Janet Roelofse (Washington, D.C., United States)

Activity Manager: Maysa Al-Khateeb (Amman, Jordan) Background of project/program/intervention (Provide a brief background on the country and/or sector context; specific problem or opportunity the intervention addresses; and the development hypothesis) HRH2030 is a global cooperative agreement consisting of both global (or ‘core’) activities, as well as country-specific activities (or ‘buy-ins’) in multiple countries, including Jordan, the first country to buy into the award and also the first to undertake a midterm evaluation. Led by Chemonics International, HRH2030 consortium members in Jordan include the Palladium Group and the University Research Co. HRH2030 works with and through Jordan’s Ministry of Health (MOH), the High Health Council (HHC), and the Civil Service Bureau (CSB), as well as other professional and civic health organization stakeholders.

HRH2030 activities in Jordan build on the decades-long partnership between the United States and Jordan in the health sector, which has contributed toward significant and tangible improvements in infant and child mortality rates and healthcare service delivery. However, Jordan’s rapidly growing population, an influx of Syrian refugees, and the decentralization of the healthcare system have intensified challenges for both health practitioners and managers. Nurses and other professionals face increased workloads, unclear career development pathways, and strained information systems. Addressing these challenges is key to fostering a stable and resilient health workforce.

Under USAID/Jordan’s Development Objective 3 (Social Sector Quality Improved), Intermediate Result 1 (Health Status Improved), USAID has emphasized its commitment to improving health sector outcomes and resilience through health systems strengthening.

Through HRH2030, USAID is helping Jordan strengthen its health workforce through interventions that advance three key results:

• Result 1: Improved Human Resources (HR) practices at the Ministry of Health (MOH) • Result 2: Improved competency of the health workforce • Result 3: Strengthened national Human Resources for Health (HRH) governance

Theory of Change of target project/program/intervention USAID/Jordan HRH2030’s theory of change is that IF the Human Resources (HR) practices and workforce competency are improved and national Human Resources for Health (HRH) governance is strengthened to provide better access and quality, THEN as a result of a strengthened health workforce, health services will be improved for the Jordanian population. In the results framework that follows below, Activity sub-results are aligned under each of the three key results that they advance. For example, improved HR practices at the MOH (Result 1) requires having trained HR individuals in place (sub-result 1.2) who are capable of developing and/or improving the HR management and HR development systems (sub-result 1.1) that cover the different HR functions including recruitment, orientation, development, planning, motivation, and succession planning.

Specific interventions under each sub-result were based on the review of outcomes in the Activity’s first two years as well as stakeholder priorities. This included integrating research topics into the applicable interventions rather than having stand-alone research interventions. Finally, crosscutting themes in the results framework include the need to address fragmentation in the health sector, decentralization efforts across Jordan’s public sector, gender, and communications to raise awareness on HRH issues in Jordan.


Strategic or Results Framework for the project/program/intervention (paste framework below)

What is the geographic coverage and/or the target groups for the project or program that is the subject of analysis? The geographic coverage is Kingdom-wide, and most of the work will be conducted in the capital Amman where the Ministry of Health and other central offices are located. The target groups are the health workforce of Jordan and the public, civic and private institutions that support them.


IX. Purpose, Audience & Application

A. Purpose: Why is this evaluation/assessment being conducted (purpose of analytic activity)? Provide the specific reason for this activity, linking it to future decisions to be made by USAID leadership, partner governments, and/or other key stakeholders.

The purpose of the evaluation is to assess the Activity’s contribution toward a strengthened health workforce for better health services. This formative evaluation should also develop recommendations for the remaining years of the Activity, both to increase effectiveness and to strengthen partnerships with government stakeholders to foster sustainability.

B. Audience: Who is the intended audience for this analysis? Who will use the results? If listing

multiple audiences, indicate which are most important.

The primary audience for this analysis is the leadership of USAID/Jordan, including the leadership of the Population and Family Health Office. Secondary audiences include the implementing partners (Chemonics and others), as well as the Government of Jordan and other stakeholders.

C. Applications and use: How will the findings be used? What future decisions will be made

based on these findings?

This formative evaluation will develop recommendations for the remaining years of the Activity, both to increase effectiveness and to strengthen partnerships with government stakeholders to foster sustainability. Also, the findings will be used to decide what if any follow-on activity should occur.

X. Evaluation/Analytic Questions & Matrix:

• Questions should be: a) aligned with the evaluation/assessment purpose and the expected use of findings; b) clearly defined to produce needed evidence and results; and c) answerable given the time and budget constraints. Include any disaggregation (e.g., sex, geographic locale, age, etc.), they must be incorporated into the evaluation/assessment questions. USAID Evaluation Policy recommends 1to 5 evaluation questions.

• State the method and/or data source and describe the data elements needed to answer the evaluation questions

Evaluation Question Method & Data Source 1 To what extent are HR practices at the

MOH improving and why? Key informant interviews; focus group discussions; survey; facility/service assessment; observations; document and data review

2 To what extent are staff receiving training and using their newly acquired skills? Why or why not?

Key informant interviews; focus group discussions; survey; facility/service assessment; observations; document and data review

3 To what extent are capacity, knowledge management, transparency, and accountability in planning, managing and retaining Jordan’s health workforce increasing? Why or why not?


4 To what extent is the Jordan HRH2030 Activity influencing women’s management and leadership in the health workforce?


5 What are the next steps for institutionalizing the new law that requires continuous professional development for Jordan’s health workforce?



Other Questions [OPTIONAL]. (Note: Use this space only if necessary. Too many questions leads to an ineffective evaluation or analysis.) PROCUREMENT-SENSITIVE EVALUATION QUESTIONS In addition to the evaluation questions listed in the body of the Statement of Work, the evaluation will address the following questions related to sustainability and any potential follow-on Activity.


2. What does the evaluation team recommend regarding any potential follow-on HRH Activity? If a follow-on is necessary, should it be similar to HRH2030, or be combined with another existing USAID/Jordan Activity, or take the form of direct support to the Government of Jordan through the utilization of partner government systems?

XI. Methods

Check and describe the recommended methods for this analytic activity. Selection of methods should be aligned with the evaluation/assessment questions and fit within the time and resources allotted for this analytic activity. Also, include the sample or sampling frame in the description of each method selected.

General Comments related to Methods: The methods described below and also in the matrix above are expected to be integrated, just as the evaluation questions themselves are integrated as HRH2030 is an intervention operating at multiple levels in the health sector in Jordan. In other words, for example, the survey will not inform just one of the evaluation questions, but likely most if not all of them. Or multiple surveys may need to be developed.

☑ Document and Data Review (list of documents and data recommended for review)

This desk review will be used to provide background information on the project/program, and will also provide data for analysis for this evaluation. Documents and data to be reviewed include:

• The awarded cooperative agreement • Jordan’s Five Year HRH2030 Approved Strategy • The approved Activity Monitoring, Evaluation and Learning Plan (AMELP), which provides

information about the Activity’s indicators for monitoring and evaluation • Quarterly, semiannual and annual reports • Activity Workplans • Activity research reports • Monitoring and Evaluation System and Dashboards • Selected project deliverables such as the MOH HR Systems and Capacity Needs Assessment

and National HRH Strategy

Some of the Activity’s contextual, activity-level and result-level performance indicators are listed below. When possible, a baseline was collected and targets were set. We do not expect these indicators to be a major focus of the evaluation since the Activity is not completed, but there will be some occasions when it will be appropriate to address ongoing progress toward targets.

a. Percentage of HR Specialized units implementing HRH management best practices, as a result of USG assistance

b. Density of health workers by 10,000 population (disaggregated by sex, cadre, and location) c. Percentage of active health workers employed by facility type and cadre d. Percentage of leadership positions in the MOH occupied by women e. Percentage of MOH staff completing in-service training courses f. Level of HRH governance strength (matrix scale) g. Percentage of training participants who reported improved knowledge and skills


h. Level of Continuous Professional Development system institutionalization (matrix scale) ☑ Key Informant Interviews (list categories of key informants, and purpose of inquiry)

USAID staff, USAID implementing partners, the Government of Jordan and other civic, public, and private health stakeholders, including members of Jordan’s health workforce and the sector beneficiaries. These interviews will inform all of the evaluation questions. Some of the key USAID/Jordan HRH2030 stakeholders and partners include:

a. Government of Jordan and its national goals and strategies, including the High Health Council (HHC), the Ministry of Health (MOH), the Civil Service Bureau (CSB), and Jordanian Royal Medical Services (JRMS)

b. Private and Civil Society Organizations including the Healthcare Accreditation Council (HCAC); the Jordan Medical Association (JMA); the Jordan Medical Council (JMC); the Jordan Nurses Association (JNA); Private Hospitals Association (PHA); and the Jordan Nursing Council (JNC)

c. Related USAID Activities include: Health Service Delivery (HSD) Activity; the construction of health facilities; Jordan Communications, Advocacy and Policy (JCAP) Activity; Jordan Health Finance and Governance (HFG) Activity; Takamol Activity; Monitoring and Evaluation Support Project (MESP) Activity; Cities Implementing Transparent, Innovative and Effective Solutions (CITIES) Activity

d. The World Health Organization’s work in Jordan e. The Organization for Economic Cooperation and Development (OECD) and the European

Union’s Support for Improvement in Governance and Management (SIGMA) f. HRH2030 beneficiaries including participants in training activities

☑ Focus Group Discussions (list categories of groups, and purpose of inquiry)

Focus groups are expected for groups of HRH2030 beneficiaries including participants in training activities. These focus groups will address all of the evaluation questions.

☑ Group Interviews (list categories of groups, and purpose of inquiry)

Key informants may be interviewed in small groups of similar respondents, as long as all participants feel free to express their own opinions.

☐ Client/Participant Satisfaction or Exit Interviews (list who is to be interviewed, and purpose of inquiry)

☑ Survey (describe content of the survey and target responders, and purpose of inquiry) One or more surveys should be developed for training activity beneficiaries (including both providers and managers), as well as for other stakeholders including selected groups within the public and civic organizations listed above

☐ Facility or Service Assessment/Survey (list type of facility or service of interest, and purpose of inquiry)

☑ Observations (list types of sites or activities to be observed, and purpose of inquiry) The evaluation team should observe a few examples of both health service provision and training provision by way of a site visit. This is a formative evaluation, and we want to see if there are any impacts at the higher levels of outcomes, even though there are still several years in the Activity.


List or describe case and counterfactual” Case Counterfactual

XII. Human Subject Protection

The Evaluation Team must develop protocols to insure privacy and confidentiality prior to any data collection. Primary data collection must include a consent process that contains the purpose of the evaluation, the risk and benefits to the respondents and community, the right to refuse to answer any question, and the right to refuse participation in the evaluation at any time without consequences. Only adults can consent as part of this evaluation. Minors cannot be respondents to any interview or survey and cannot participate in a focus group discussion without going through an IRB. The only time minors can be observed as part of this evaluation is as part of a large community-wide public event, when they are part of family and community in the public setting. During the process of this evaluation, if data are abstracted from existing documents that include unique identifiers, data can only be abstracted without this identifying information. An Informed Consent statement included in all data collection interactions must contain:

• Introduction of facilitator/note-taker • Purpose of the evaluation/assessment • Purpose of interview/discussion/survey • Statement that all information provided is confidential and information provided will not be

connected to the individual • Right to refuse to answer questions or participate in interview/discussion/survey • Request consent prior to initiating data collection (i.e., interview/discussion/survey)

XIII. Analytic Plan

Describe how the quantitative and qualitative data will be analyzed. Include method or type of analyses, statistical tests, and what data it to be triangulated (if appropriate). For example, a thematic analysis of qualitative interview data, or a descriptive analysis of quantitative survey data. All analyses will be geared to answer the evaluation questions. Additionally, the evaluation will review both qualitative and quantitative data related to the project/program’s achievements against its objectives and/or targets.

Quantitative data will be analyzed primarily using descriptive statistics. Data will be stratified by demographic characteristics, such as sex, age, and location, whenever feasible. Other statistical test of association (i.e., odds ratio) and correlations will be run as appropriate.

Thematic review of qualitative data will be performed, connecting the data to the evaluation questions, seeking relationships, context, interpretation, nuances and homogeneity and outliers to better explain what is happening and the perception of those involved. Qualitative data will be used to substantiate quantitative findings, provide more insights than quantitative data can provide, and answer questions where other data do not exist.

Use of multiple methods that are quantitative and qualitative, as well as existing data (e.g., project/program performance indicator data, DHS, MICS, HMIS data, etc.) will allow the Team to triangulate findings to produce more robust evaluation results.

The Evaluation Report will describe analytic methods and statistical tests employed in this evaluation.


XIV. Activities

List the expected activities, such as Team Planning Meeting (TPM), briefings, verification workshop with IPs and stakeholders, etc. Activities and Deliverables may overlap. Give as much detail as possible. Background reading – Several documents are available for review for this analytic activity. These include the HRH2030 cooperative agreement, annual work plans, M&E plans, quarterly progress reports, and routine reports of project performance indicator data, as well as survey data reports (i.e., DHS). This desk review will provide background information for the Evaluation Team, and will also be used as data input and evidence for the evaluation. Team Planning Meeting (TPM) – A four-day team planning meeting (TPM) will be held at the initiation of this assignment and before the data collection begins. The TPM will:

• Review and clarify any questions on the evaluation SOW • Clarify team members’ roles and responsibilities • Establish a team atmosphere, share individual working styles, and agree on procedures for

resolving differences of opinion • Review and finalize evaluation questions • Review and finalize the assignment timeline • Develop data collection methods, instruments, tools and guidelines • Review and clarify any logistical and administrative procedures for the assignment • Develop a data collection plan • Draft the evaluation work plan for USAID’s approval • Develop a preliminary draft outline of the team’s report • Assign drafting/writing responsibilities for the final report

Briefing and Debriefing Meetings – Throughout the evaluation the Team Lead will provide briefings to USAID. The In-Brief and Debrief are likely to include the all Evaluation Team experts, but will be determined in consultation with the Mission. These briefings are:

• Evaluation launch, a call/meeting among the USAID, GH Pro and the Team Lead to initiate the evaluation activity and review expectations. USAID will review the purpose, expectations, and agenda of the assignment. GH Pro will introduce the Team Lead, and review the initial schedule and review other management issues.

• In-brief with USAID/Jordan, as part of the TPM. At the beginning of the TPM, the Evaluation Team will meet with USAID to discuss expectations, review evaluation questions, and intended plans. The Team will also raise questions that they may have about the project/program and SOW resulting from their background document review. The time and place for this in-brief will be determined between the Team Lead and USAID prior to the TPM.

• Workplan and methodology review briefing. At the end of the TPM, the Evaluation Team will meet with USAID/Jordan to present an outline of the methods/protocols, timeline and data collection tools. Also, the format and content of the Evaluation report(s) will be discussed.

• In-brief with project to review the evaluation plans and timeline, and for the project to give an overview of the project to the Evaluation Team.

• The Team Lead (TL) will brief the USAID/Jordan weekly to discuss progress on the evaluation. As preliminary findings arise, the TL will share these during the routine briefing, and in an email.

• A final debrief between the Evaluation Team and USAID/Jordan will be held at the end of the evaluation to present preliminary findings to USAID. During this meeting a summary of the data will be presented, along with high level findings and draft recommendations. For the


debrief, the Evaluation Team will prepare a PowerPoint Presentation of the key findings, issues, and recommendations. The evaluation team shall incorporate comments received from USAID during the debrief in the evaluation report. (Note: preliminary findings are not final and as more data sources are developed and analyzed these finding may change.)

• IP and Stakeholders’ debrief/workshop will be held with the project staff and other stakeholders identified by USAID. This will occur following the final debrief with the Mission, and will not include any information that may be procurement deemed sensitive or not suitable by USAID. Most likely the IP and the MOH will participate in this debrief, but this will be discussed during the in-brief with USAID to determine who will attend.

Fieldwork, Site Visits and Data Collection – The evaluation team will conduct site visits to for data collection. Selection of sites to be visited will be finalized during TPM in consultation with USAID. The evaluation team will outline and schedule key meetings and site visits prior to departing to the field. Evaluation/Analytic Report – The Evaluation/Analytic Team under the leadership of the Team Lead will develop a report with findings and recommendations (see Analytic Report below). Report writing and submission will include the following steps:

1. Team Lead will submit draft evaluation report to GH Pro for review and formatting 2. GH Pro will submit the draft report to USAID 3. USAID will review the draft report in a timely manner, and send their comments and edits

back to GH Pro 4. USAID will manage implementing partner(s)’s (IP) review of the report and compile and send

their comments and edits to GH Pro. (Note: USAID will decide what draft they want the IP to review.)

5. GH Pro will share USAID’s comments and edits with the Team Lead, who will then do final edits, as needed, and resubmit to GH Pro

6. GH Pro will review and reformat the final Evaluation/Analytic Report, as needed, and resubmit to USAID for approval.

7. Once Evaluation Report is approved, GH Pro will re-format it for 508-compliance and post it to the DEC.

The Evaluation Report excludes any procurement-sensitive and other sensitive but unclassified (SBU) information. This information will be submitted in a memo to USIAD separate from the Evaluation Report. Data Submission – All quantitative data will be submitted to GH Pro in a machine-readable format (CSV or XML). The datasets created as part of this evaluation/assessment must be accompanied by a data dictionary that includes a codebook and any other information needed for others to use these data. It is essential that the datasets are stripped of all identifying information, as the data will be public once posted on USAID Development Data Library (DDL). Where feasible, qualitative data that do not contain identifying information should also be submitted to GH Pro.

XV. Deliverables and Products

Select all deliverables and products required on this analytic activity. For those not listed, add rows as needed or enter them under “Other” in the table below. Provide timelines and deliverable deadlines for each. Deliverable / Product Timelines & Deadlines (estimated) ☑ Launch briefing August 6, 2018 ☑ In-brief with USAID August 29, 2018


Deliverable / Product Timelines & Deadlines (estimated) ☑ Workplan and methodology review briefing September 2, 2018 ☑ Workplan (must include questions, methods, timeline, data analysis plan, and instruments)

August 31, 2018

☑ In-brief with target project / program September 2, 2018 ☑ Routine briefings Weekly ☑ Debrief with USAID with Power Point presentation

September 30, 2018

☑ IP & stakeholders findings review workshop with Power Point presentation

October 1, 2018

☑ Draft report Submit to GH Pro: October 24, 2018 GH Pro submits to USAID: October 31, 2018 USAID provides feedback: November 27, 2018

☑ Final report Submit to GH Pro: December 4, 2018 GH Pro submits to USAID: December 11, 2018 USAID approves technical content: January 6, 2018

☑ Raw data (cleaned datasets in CSV or XML with codesheet)

December 12, 2018

☑ Report Posted to the DEC February 8, 2019 ☐ Other (specify): Holidays:

August 22, 2018 .................... Eid al-Adha ........................... Jordan August 23, 2018 .................... Eid al-Adha ........................... Jordan August 24, 2018 .................... Eid al-Adha ........................... Jordan August 25, 2018 .................... Eid al-Adha ........................... Jordan September 3, 2018................ Labor Day ............................ US September 12, 2018 ............. Muharram/New Year ........ Jordan October 8, 2018.................... Columbus Day .................... US November 12, 2018 ............. Veterans Day ...................... US November 21, 2018 ............. Prophet's Birthday ............. Jordan November 22, 2018 ............. Thanksgiving ........................ US December 25, 2018 .............. Christmas

Estimated USAID review time Average number of business days USAID will need to review the Report? 15 Business days

XVI. Team Composition, Skills and Level Of Effort (LOE)

Evaluation/Assessment team: When planning this analytic activity, consider: • Key staff should have methodological and/or technical expertise, regional or country experience,

language skills, team lead experience and management skills, etc. • Team leaders for evaluations/assessments must be an external expert with appropriate skills and

experience. • Additional team members can include research assistants, enumerators, translators, logisticians,

etc. • Teams should include a collective mix of appropriate methodological and subject matter

expertise. • Evaluations require an Evaluation Specialist, who should have evaluation methodological

expertise needed for this activity. Similarly, other analytic activities should have a specialist with methodological expertise.

https://www.timeanddate.com/holidays/jordan/eid-al-adha





• Note that all team members will be required to provide a signed statement attesting that they have no conflict of interest (COI), or describing the conflict of interest if applicable.

Team Qualifications: Please list technical areas of expertise required for this activity:

• List desired qualifications for the team as a whole • List the key staff needed for this analytic activity and their roles. • Sample position descriptions are posted on USAID/GH Pro webpage • Edit as needed GH Pro provided position descriptions

OVERALL TEAM REQUIREMENTS All proposed team members will be required to submit resumes and signed written disclosures of any conflicts of interest. As a whole the team should include the following skillsets and characteristics:

1. Between two to four total team members 2. Team Leader (outside expert) with at least ten years of overall relevant experience including

five years of experience with USAID reporting requirements in the health and/or evaluation sectors

3. At least one team member (evaluation specialist) with five or more years of USAID evaluation experience, especially in the health sector

4. At least one native-level Arabic speaker (two preferred) 5. Across all team members:

a. Combined three or more years of experience in Jordan’s health sector b. Combined three or more years of Human Resources for Health experience c. One post-graduate degree in public health, health systems, or organizational development

Team Lead: This person will be selected from among the key staff, and will meet the requirements of both this and the other position. The team lead should have significant experience conducting project evaluations and/or assessments.

Roles & Responsibilities: The team leader will be responsible for (1) providing team leadership; (2) managing the team’s activities, (3) ensuring that all deliverables are met in a timely manner, (4) serving as a liaison between the USAID and the evaluation/assessment team, and (5) leading briefings and presentations.

Qualifications: • Minimum of 10 years of experience in public health, which included experience in

implementation of health activities in developing countries • Demonstrated experience leading health sector project/program evaluation/assessments,

utilizing both quantitative and qualitative s methods • Excellent skills in planning, facilitation, and consensus building • Excellent interpersonal skills, including experience successfully interacting with host government

officials, civil society partners, and other stakeholders • Excellent skills in project management • Excellent organizational skills and ability to keep to a timeline • Good writing skills, with extensive report writing experience • Experience working in the region, and experience in Jordan is desirable • Familiarity with USAID • Familiarity with USAID policies and practices

− Evaluation policy − Results frameworks


− Performance monitoring plans Key Staff 2 Title: Human Resources for Health Specialist (2 consultants)

Roles & Responsibilities: Serve as a member of the evaluation team, providing expertise in human resources for health. S/He will participate in planning and briefing meetings, data collection, data analysis, development of evaluation presentations, and writing of the Evaluation Report. Qualifications:

• At least 5 years’ experience with human resources for health projects; USAID project implementation experience preferred

• Familiar with theory and practice to improve HRH capacity, competency and governance at a local and national level in low resourced, decentralized and challenged health care systems/environments

• Excellent interpersonal skills, including experience successfully interacting with host government officials, civil society partners, and other stakeholders

• Proficient in English; and Arabic preferred but not required • Experience working in the region, and experience in Jordan is desirable • Good writing skills, including experience writing evaluation and/or assessment reports • Experience in conducting USAID evaluations of health programs/activities

Other Staff Titles with Roles & Responsibilities (include number of individuals needed):

Local Evaluators (2 consultants) to assist the Evaluation Team with data collection, analysis and data interpretation. They will have basic familiarity with health topics, as well as experience conducting surveys interviews and focus group discussion, both facilitating and note taking. Furthermore, they will assist in translation of data collection tools and transcripts, as needed. The Local Evaluators will have a good command of English and Arabic. They will also assist the Team and, as needed. They will report to the Team Lead.

Will USAID participate as an active team member or designate other key stakeholders to as an active team member? This will require full time commitment during the evaluation or assessment activity.

☐ Full member of the Evaluation Team (including planning, data collection, analysis and report development) – If yes, specify who: ☐ Some Involvement anticipated – If yes, specify who: ☑ No

Staffing Level of Effort (LOE) Matrix:

This LOE Matrix will help you estimate the LOE needed to implement this analytic activity. If you are unsure, GH Pro can assist you to complete this table.

a) For each column, replace the label "Position Title" with the actual position title of staff needed for this analytic activity.

b) Immediately below each staff title enter the anticipated number of people for each titled position.

c) Enter Row labels for each activity, task and deliverable needed to implement this analytic activity.

d) Then enter the LOE (estimated number of days) for each activity/task/deliverable corresponding to each titled position.


e) At the bottom of the table total the LOE days for each consultant title in the ‘Sub-Total’ cell, then multiply the subtotals in each column by the number of individuals that will hold this title.

Level of Effort in days for each Evaluation/Analytic Team member (The following is an Illustrative LOE Chart. Please edit to meet the requirements of this activity.)

Activity / Deliverable Evaluation/Analytic Team

Team Lead HRH Specialist

Local Evaluators

Number of persons → 1 2 2 1 Launch Briefing 0.5 2 Desk review 5 5 5 3 Preparation for Team convening in-country 0.5 4 Travel to country 2 2 5 In-brief with Mission 0.5 0.5 0.5 6 Team Planning Meeting 4 4 4 7 Workplan and methodology briefing with USAID 0.5 0.5 0.5 8 Eval planning deliverables: 1) workplan with

timeline, eval matrix, protocol (methods, sampling & analytic plan); 2) data collection tools

9 In-brief with HRH2030 0.5 0.5 0.5 10 Data Collection DQA Workshop (protocol

orientation/training for all data collectors) 2 2 2

11 Prep / Logistics for Site Visits 0.5 0.5 0.5 12 Data collection / Site Visits (including travel to

sites) 20 20 20

13 Data analysis 5 5 5 14 Debrief with Mission with prep 1 1 1 15 IP & Stakeholder debrief workshop with prep 1 1 1 16 Depart country 2 2 17 Draft report(s) 8 7 5 18 GH Pro Report QC Review & Formatting 19 Submission of draft report(s) to Mission 20 USAID Report Review 21 Revise report(s) per USAID comments 4 3 1 22 Finalize and submit report to USAID 23 USAID approves report 24 Final copy editing and formatting 25 508 Compliance editing 26 Eval Report(s) to the DEC Total LOE per person 57 54 46 Total LOE 57 54 92

If overseas, is a 6-day workweek permitted, as needed ☑ Yes ☐ No

Travel anticipated: List international and local travel anticipated by what team members.


Most evaluation activities will be within a same-day travel driving distance of Amman. Some work may require overnight stay in Aqaba in southern Jordan.

XVII. Logistics

Visa Requirements List any specific Visa requirements or considerations for entry to countries that will be visited by consultant(s):

US Citizens can get a visa upon arrival. GH Pro will verify visa requirement for other nationalities, as needed.

List recommended/required type of Visa for entry into counties where consultant(s) will work

Name of Country Type of Visa Jordan ☑ Tourist ☐ Business ☐ No preference ☐ Tourist ☐ Business ☐ No preference ☐ Tourist ☐ Business ☐ No preference

Clearances & Other Requirements Note: Most Evaluation/Analytic Teams arrange their own work space, often in conference rooms at their hotels. However, if a Security Clearance or Facility Access is preferred, GH Pro can submit an application for it on the consultant’s behalf.

GH Pro can obtain Facility Access (FA) and transfer existing Secret Security Clearance for our consultants, but please note these requests, processed through AMS at USAID/GH (Washington, DC), can take 4-6 months to be granted. If you are in a Mission and the RSO is able to grant a temporary FA locally, this can expedite the process. FAs for non-US citizens or Green Card holders must be obtained through the RSO. If FA or Security Clearance is granted through Washington, DC, the consultant must pick up his/her badge in person at the Office of Security in Washington, DC, regardless of where the consultant resides or will work.

If Electronic Country Clearance (eCC) is required prior to the consultant’s travel, the consultant is also required to complete the High Threat Security Overseas Seminar (HTSOS). HTSOS is an interactive e-Learning (online) course designed to provide participants with threat and situational awareness training against criminal and terrorist attacks while working in high threat regions. There is a small fee required to register for this course. [Note: The course is not required for employees who have taken FACT training within the past five years or have taken HTSOS within the same calendar year.]

If eCC is required, and the consultant is expected to work in country more than 45 consecutive days, the consultant may be required complete the one week Foreign Affairs Counter Threat (FACT) course offered by FSI in West Virginia. This course provides participants with the knowledge and skills to better prepare themselves for living and working in critical and high threat overseas environments. Registration for this course is complicated by high demand (consultants must register approximately 3-4 months in advance). Additionally, there will be the cost for additional lodging and M&IE to take this course.

Check all that the consultant will need to perform this assignment, including USAID Facility Access, GH Pro workspace and travel (other than to and from post).

☐ USAID Facility Access (FA) Specify who will require Facility Access:


☐ Electronic County Clearance (ECC) (International travelers only) ☐ High Threat Security Overseas Seminar (HTSOS) (required in most countries with ECC) ☐ Foreign Affairs Counter Threat (FACT) (for consultants working on country more than 45 consecutive days)

☐ GH Pro workspace Specify who will require workspace at GH Pro:

☑ Travel -other than posting (specify): Travel within Jordan ☐ Other (specify):

Specify any country-specific security concerns and/or requirements

GH Pro will review and monitor State Department official travel website. The security environment in Jordan requires vigilance, as it is in the middle of a volatile region, and there are ongoing threats within and around Jordan.

XVIII. GH PRO Roles and Responsibilities

GH Pro will coordinate and manage the evaluation/assessment team and provide quality assurance oversight, including:

• Review SOW and recommend revisions as needed • Provide technical assistance on methodology, as needed • Develop budget for analytic activity • Recruit and hire the evaluation/assessment team, with USAID POC approval • Arrange international travel and lodging for international consultants • Request for country clearance and/or facility access (if needed) • Review methods, workplan, analytic instruments, reports and other deliverables as part of the

quality assurance oversight • Report production - If the report is public, then coordination of draft and finalization steps,

editing/formatting, 508ing required in addition to and submission to the DEC and posting on GH Pro website. If the report is internal, then copy editing/formatting for internal distribution.

XIX. USAID Roles and Responsibilities

Below is the standard list of USAID’s roles and responsibilities. Add other roles and responsibilities as appropriate.

USAID Roles and Responsibilities USAID will provide overall technical leadership and direction for the analytic team throughout the assignment and will provide assistance with the following tasks: Before Field Work • SOW.

◦ Develop SOW. ◦ Peer Review SOW ◦ Respond to queries about the SOW and/or the assignment at large.

• Consultant Conflict of Interest (COI). To avoid conflicts of interest or the appearance of a COI, review previous employers listed on the CV’s for proposed consultants and provide additional information regarding potential COI with the project contractors evaluated/assessed and information regarding their affiliates.

• Documents. Identify and prioritize background materials for the consultants and provide them to GH Pro, preferably in electronic form, at least one week prior to the inception of the assignment.


• Local Consultants. Assist with identification of potential local consultants, including contact information.

• Site Visit Preparations. Provide a list of site visit locations, key contacts, and suggested length of visit for use in planning in-country travel and accurate estimation of country travel line items costs.

• Lodgings and Travel. Provide guidance on recommended secure hotels and methods of in-country travel (i.e., car rental companies and other means of transportation).

During Field Work • Mission Point of Contact. Throughout the in-country work, ensure constant availability of the

Point of Contact person and provide technical leadership and direction for the team’s work. • Meeting Space. Provide guidance on the team’s selection of a meeting space for interviews and/or

focus group discussions (i.e. USAID space if available, or other known office/hotel meeting space). • Meeting Arrangements. Assist the team in arranging and coordinating meetings with stakeholders. • Facilitate Contact with Implementing Partners. Introduce the analytic team to implementing

partners and other stakeholders, and where applicable and appropriate prepare and send out an introduction letter for team’s arrival and/or anticipated meetings.

After Field Work • Timely Reviews. Provide timely review of draft/final reports and approval of deliverables.

XX. Analytic Report

Provide any desired guidance or specifications for Final Report. (See How-To Note: Preparing Evaluation Reports)

The Evaluation/Analytic Final Report must follow USAID’s Criteria to Ensure the Quality of the Evaluation Report (found in Appendix I of the USAID Evaluation Policy).

• The report should not exceed 15 pages (excluding executive summary, table of contents, acronym list and annexes).

• The Executive Summary will be translated into Arabic (two versions: English and Arabic). • The structure of the report should follow the Evaluation Report template, including branding

found here or here. • Draft reports must be provided electronically, in English, to GH Pro who will then submit it

to USAID. • For additional Guidance, please see the Evaluation Reports to the How-To Note on preparing

Evaluation Draft Reports found here. USAID Criteria to Ensure the Quality of the Evaluation Report (USAID ADS 201):

• Evaluation reports should be readily understood and should identify key points clearly, distinctly, and succinctly.

• The Executive Summary of an evaluation report should present a concise and accurate statement of the most critical elements of the report.

• Evaluation reports should adequately address all evaluation questions included in the SOW, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID.

• Evaluation methodology should be explained in detail and sources of information properly identified.

• Limitations to the evaluation should be adequately disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.).

• Evaluation findings should be presented as analyzed facts, evidence, and data and not

http://www.usaid.gov/sites/default/files/documents/1870/How-to-Note_Preparing-Evaluation-Reports.pdf


http://www.usaid.gov/evaluation/policy

http://usaidlearninglab.org/library/evaluation-report-template

http://usaidprojectstarter.org/content/usaid-evaluation-report-template


https://www.usaid.gov/sites/default/files/documents/1870/201.pdf


based on anecdotes, hearsay, or simply the compilation of people’s opinions. • Findings and conclusions should be specific, concise, and supported by strong quantitative

or qualitative evidence. • If evaluation findings assess person-level outcomes or impact, they should also be

separately assessed for both males and females. • If recommendations are included, they should be supported by a specific set of findings

and should be action-oriented, practical, and specific. Reporting Guidelines: The draft report should be a comprehensive analytical evidence-based evaluation/assessment report. It should detail and describe results, effects, constraints, and lessons learned, and provide recommendations and identify key questions for future consideration. The report shall follow USAID branding procedures. The report will be edited/formatted and made 508 compliant as required by USAID for public reports and will be posted to the USAID/DEC. The findings from the evaluation/assessment will be presented in a draft report at a full briefing with USAID and at a follow-up meeting with key stakeholders. The report should use the following format:

• Abstract: briefly describing what was evaluated, evaluation questions, methods, and key findings or conclusions (not more than 250 words)

• Executive Summary: summarizes key points, including the purpose, background, evaluation questions, methods, limitations, findings, conclusions, and most salient recommendations (2-5 pages)

• Table of Contents (1 page) • Acronyms • Evaluation/Analytic Purpose and Evaluation/Analytic Questions: state purpose of, audience for,

and anticipated use(s) of the evaluation/assessment (1-2 pages) • Project [or Program] Background: describe the project/program and the background ,

including country and sector context, and how the project/program addresses a problem or opportunity (1-3 pages)

• Evaluation/Analytic Methods and Limitations: data collection, sampling, data analysis and limitations (1-3 pages)

• Findings (organized by Evaluation/Analytic Questions): substantiate findings with evidence/data • Conclusions • Recommendations • Annexes

◦ Annex I: Evaluation/Analytic Statement of Work ◦ Annex II: Evaluation/Analytic Methods and Limitations ((if not described in full in the

main body of the evaluation report) ◦ Annex III: Data Collection Instruments ◦ Annex IV: Sources of Information

• List of Persons Interviews • Bibliography of Documents Reviewed • Databases • [etc.]

◦ Annex V: Statement of Differences (if applicable) ◦ Annex VI: Disclosure of Any Conflicts of Interest ◦ Annex VII: Summary information about evaluation team members, including

qualifications, experience, and role on the team. The evaluation methodology and report will be compliant with the USAID Evaluation Policy and Checklist for Assessing USAID Evaluation Reports

http://www.usaid.gov/sites/default/files/documents/2151/USAIDEvaluationPolicy.pdf

http://usaidlearninglab.org/sites/default/files/resource/files/mod11_summary_checklist_for_assessing_usaid_evaluation_reports.pdf


-------------------------------- The Evaluation Report should exclude any potentially procurement-sensitive information. As needed, any procurement sensitive information or other sensitive but unclassified (SBU) information will be submitted in a memo to USIAD separate from the Evaluation Report. -------------------------------- All data instruments, data sets (if appropriate), presentations, meeting notes and report for this evaluation/analysis will be submitted electronically to the GH Pro Program Manager. All datasets developed as part of this evaluation activity will be submitted to GH Pro in an unlocked machine-readable format (CSV or XML). The datasets must not include any identifying or confidential information. The datasets must also be accompanied by a data dictionary that includes a codebook and any other information needed for others to use these data. Qualitative data included in this submission should not contain identifying or confidential information. Category of respondent is acceptable, but names, addresses and other confidential information that can easily lead to identifying the respondent should not be included in any quantitative or qualitative data submitted.

XXI. USAID Contacts

Primary Contact Alternate Contact 1 Alternate Contact 2 Name: Rand Milhem Andrea Halverson Ruba Al-kalouti Title: Deputy Director USAID Mission

Program Office USAID/Jordan

Health Office USAID/Jordan

USAID/Jordan

Email: [email protected] [email protected] [email protected] Telephone: Cell Phone:

List other contacts who will be supporting the Requesting Team with technical support, such as reviewing SOW and Report (such as USAID/W GH Pro management team staff)

Technical Support Contact 1 Technical Support Contact 2 Name: Janet Roelofse Title: USAID Office: USAID/GH/OHS Email: Telephone: Cell Phone:

XXII. Other Reference Materials

Documents and materials needed and/or useful for consultant assignment, that are not listed above

XXIII. Adjustments Made in Carrying out This SOW after Approval of the SOW (To be

completed after Assignment Implementation by GH Pro)

mailto:[email protected]

mailto:[email protected]


ANNEX II. DATA COLLECTION INSTRUMENTS

KII QUESTION GUIDES

Introduction for Key Informants Interviews

Interviewer: please read this introduction and obtain the approval of the respondent to your question about consent before proceeding with the interview.

My name is….. and along with my colleagues from the team ….. (Name or names) I would like to thank you very much for your willingness to talk to us. The aim of our conversation is to get a better idea about human resources in the health sector in Jordan and to investigate challenges and opportunities for health workers. This conversation today is part of a larger study/evaluation that looks at the last three years of activities related to policies and programs in human resources for health in Jordan, and we are talking to various stakeholders. We are conducting similar in-depth interviews with many officials and professionals from various sectors and organizations related to the subject of human resources for health.

We would like to have an open and honest conversation with you about different aspects of working in the health sector. We are here to listen to you and learn from you. We have prepared a number of topics on which we will ask your opinion and we are happy to share a copy of the questions we have prepared if you wish (give hand out if needed).

It is very important that you feel no restraints to speaking your mind. This is totally voluntary, and you can choose to not answer any question, or to end the interview at any time, with no negative repercussions.

Be as open, direct and sincere as you can. The statements you make should be based on real-life experiences and observations. It is important to us to know how the situation really is and not how the situation ought to be, unless we ask you otherwise.

We would like to stress that your personal contributions and views will not be shared with anyone outside this room.

Our conversation will be recorded on paper, on tape or both. This is because we do not want to miss anything of what you say. However, nothing of what you say will ever be made public with a name attached. The results will only be cited as a general category in the report – for example, a discussion with “a selected group of health professionals and workers or institutions,” or “an interview with a health care provider.” Our interview may last up to an hour and we thank you in advance of giving us some of your valuable time

Do you have any questions? Do we have your permission to begin?


Note: All KIIs must begin with introduction and oral consent, see document #1

KII questions for HRH 2030 management and technical staff

• Workplan questions

• How many of the MOH HRH gaps identified in the needs assessment and research are being addressed by HRH2030?

Questions about approaches

• In the list of assumptions presented in the AMELP, which ones are so far holding and which others are not, if any?

• How would you characterize the project approach to working with the MOH and other partners?

• How are disagreements and potential or actual conflicts identified and dealt with?

QR and AR questions

• In looking at your achievements to date, what interventions do you feel were accomplished with the least obstacles or unforeseen problems and surprises? Can you share with us a few of the most important lessons learned from the execution of the project in the last two years, both in terms of what you would consider positive and negative? If you have success stories that were shared or new ones not yet shared, could you share some of these with us?

Questions about the planning process

• How much involvement and engagement of partners and beneficiaries took place?

• What needs assessments, stakeholders’ workshops, consultations, other events were most useful to you to complete your approach strategy and plans?

• What measures did you have in place to ensure engagement, participation, and ownership?

• What obstacles did you face in the above processes?

• The year 2 annual report stated challenges in working with other USAID activities. What were those challenges? What did USAID do or not do to facilitate synergies and cooperation?

Questions about specific areas of project activities

Training - please give an idea of how satisfied you are with the various processes in developing and conducting your training activities: • deciding on topics for training

• scheduling

• selecting participants

• convening events

• quality of the trainers

• quality of the training materials

• evaluation of the training and post training follow up if any


• In your reports, the number of trainees (813) and the number of hours (3851) give an average of 4 hours per person. What proportion of the activities were orientations versus skill training?

• Were there constraints that forced you to allocate shorter time to training than you would have liked?

• Were those constraints discussed with your counterparts, partners, and USAID?

• What measures did you take to mitigate these constraints and ensure quality?

• Do you feel the quality of training has been at all compromised by these or any other factors?

• Can you share with us your plans for institutionalizing training in HRH?

• Does HRH2030 use local training institutions to conduct training, or does it hire independent local trainers, use its own team of trainers or has it developed an MOH team of trainers either at national or directorate level?

• Were there manuals and handbooks produced? Other tools or handouts?

Development of the HRH strategy:

• How were the targets set in the HRH strategy?

• Of the strategy priorities and targets, which ones seem to be closer to the objectives of this project and to the objectives of other USAID or other donor projects, and which ones are not on any donor’s radar?

• Will the implementation plan allocate roles to various donors?

• You state in the AR that the HRH2030 presented the HRH strategy to the Minister. Whom would you identify as the entity, group or individual from the Jordanian side who was your main partner in developing this strategy?

• Why was this counterpart not the one to present the strategy as the product of a national group rather than that of an external project?

• You mentioned in the AR YR2 that the implementation plan has been delayed and MOH did not concur with the urgency of issuing the plan. Could this be a symptom of feeling lack of ownership and engagement on the part of the MOH?

Assessments and studies:

• Did the HRH2030 project studies and assessments make use of local capacities and institutions

• Is the capacity development of such institutions part of a deliberate plan at HRH2030 to improve local research resources?

Support to the MOH:

• How does HRH 2030 channel its technical and management support, if any, to the MOH - especially the HRD and HRM functions?

• What human resources from the project are allocated to work within the MOH?

• What institutionalized mechanisms exist for regular consultations and joint planning? (Technical working groups, advisory groups, steering committees etc.)


• What mechanisms are used by the project to make financial resources available directly or indirectly to the MOH for implementing priority activities (grants, payment of costs of activities, in kind donations, etc.).

• Has HRH2030 conducted any form of assessments to gauge perceptions and opinions of the MOH managers and staff with regard to HRH2030 activities and to what extent they are understood by the organization, and/or acceptable within the organization’s culture?

Future directions and plans

• When do you think the action plan for operationalizing and monitoring the implementation of the National HRH Strategy will be finalized and approved for use?

• What do you think would be the implications of reorganizing the HHC to be within the MOH structure on its role in implementing the National HRH Strategy?

• Does HRH2030 plan to support the MOH in developing staffing plans based on the piloting of WISN?

• How does the project plan to support the MOH in developing a need-based education and training plan for different categories of service providers?

• To what extent has the HRH2030 Activity engaged the RMS HR Department in its initiatives?

• How does HRH2030 plan to work with stakeholders to initiate positive change in the CSB practices in relation to HRH recruitment, promotion and retirement?


KII-Private Sector Stakeholders

Interviewer: please read the standard introduction and obtain the approval of the respondent to your question about consent before proceeding with the interview.

• How well do you think public-private partnership in Jordan health sector is functioning?

a. Probe: Are the private health sector and related civil society organizations adequately involved in health sector development and reform initiatives?

b. Probe: Any suggested action or modalities for improving that?

• How familiar are you with the initiatives of the USAID HRH2030 Project to improve HRH functions?

a. Probe: Do any of them have the potential to improve HRH functions in the public and private health sectors?

• Do you think the HRH2030 has been successful in engaging private health sector representatives, health professional associations and regulatory bodies in decision-making forums (working group and taskforce) supporting the planning and implementation of Project’s initiatives?

a. Probe: Any recommended action to HRH2030 for improving key stakeholder’s engagement?

• Do you think the bylaws for mandatory health professionals license renewal will lead to a Continuing Professional Development (CPD) system with adequate training opportunities and use of its outcome in performance management and promotion?

a. Probe: Any recommended action to HRH2030 to achieve that?

• Do you think the HRH2030 support to the National HRH Observatory has the potential to ensure the availability of the needed gender segregated HRH data and evidence for decision-making use in the public and private health sector?

a. Probe: Any recommended action to HRH2030 to support achieving that?

• Do you think the HRH2030 support for establishing a coalition for a women’s health leadership network has the potential to be fruitful?


• How can the findings of HRH2030 supported research on “examining barriers and enablers of women’s career progression into health management positions” and those of “the motivation and retention of health workers” study be useful in improving the related HRH practices in the public and private health sector?



KII of government officials other than MOH ones


Questions

• Describe what you know about the initiatives of the HRH2030 Activity Project to improve HRH functions?

a. Probe: Which ones do you participate in, and to what degree do they have the potential to improve HRH practices?

• Do you think the engagement of HRH2030 with stakeholders, including the Higher Health Council (HHC) and Civil Service Bureau (CSB), Royal Medical Services, Jordan professional councils and associations, has been wide and effective?

a. Probe: Any recommended action to HRH2030 for improving its stakeholder engagement?

• In your view, has the developed National HRH Strategy adequately addressed national HRH priorities?

a. Probe: Gender considerations

b. Probe: Ensures the most vulnerable populations, especially poor women and children receive essential, quality services?

c. Probe: Any recommendation to HRH2030 to support achieving that in the implementation plan?

• Do you think the bylaws for mandatory health professionals license renewal will lead to a Continuing Professional Development (CPD) system with adequate training opportunities?

a. Probe: Will it affect performance management and promotion?

b. Probe: Any recommended action to HRH2030 to achieve that?

c. Probe: Will it improve medical care?

• Does HRH2030’s support to the National HRH Observatory have the potential to ensure the availability of necessary gender segregated HRH data for decision-making and to strengthen HRH functions?


• Do you think the plan for implementing the workload indicators of staffing need (WISN) tool, could be expanded to other government health services?

a. Probe: Could it be used as a step towards developing a need-based National HRH Plan?

b. Probe: Any recommended action to HRH2030 to support that?

• Does HRH2030 support for establishing a coalition for a women’s health leadership network have the potential to be fruitful?

a. Probe: Any recommended action to improve support to the network?

• What potential do the findings of HRH2030 supported research on “examining barriers and enablers of women’s career progression into health management positions” and those of “the motivation and retention of health workers” study have for improving the related HRH practices?

a. Probe: Any recommended action to HRH2030 to ensure they have impact?


KII Questions for MOH Senior Level Managers


• What do you know about the initiatives of the HRH2030 Activity Project to improve HRH functions?

a. Probe: Do you think these initiatives are in line with the MOH priorities?

b. Probe: Which do you participate in, and to what degree do they have the potential to improve HRH practices?

• Do you think the engagement of HRH2030 with stakeholders, including the use the Human Resources for Health Technical Working Group (HRH TWG), has been wide and effective?

a. Probe: Any recommended action to HRH2030 for improving its stakeholder engagement?

• Do you think that HRH2030 supported capacity building of HR staff (HRM&HRD) at the MOH central, health directorates and facility levels has the potential to strengthen their capacity and improving their performance?

a. Probe: Will it be more effective at some levels than at others?

b. Probe: Any recommended action to HRH2030 to improve the program?

• In your view, has the developed National HRH Strategy adequately addressed gender considerations and ensured the most vulnerable populations, especially poor women and children, receive the necessary, quality services?

a. Probe: What support from HRH2030 is required to achieve the strategy?

• Are the competency-based job descriptions and performance management checklists developed with HRH2030 support likely to be used in staff selection, capacity building and performance management?

a. Probe: Any recommended action to HRH2030 for making better use of them?

• Will the bylaws for mandatory health professionals license renewal lead to a CPD system with adequate training opportunities and its use in performance management and promotion?

a. Probe: Any recommended action to HRH2030 to help achieve that?

• Do you think the HRH2030 supported training on HR management and leadership has the potential to improve leadership and HR practices and increase women’s participation in leadership roles?

a. Probe: Any recommended action to HRH2030 to make the future training more inclusive and effective?

• In your opinion, is the plan for implementing the workload indicators of staffing need (WISN) tool, developed with HRH2030 support, being used by the MOH in workforce planning?

a. Probe: Any recommended action to HRH2030 to ensure better use of it?

• There was a recent “the motivation and retention of health workers” study, conducted and disseminated with HRH2030 support. What do you know about it?

a. Probe: Do you think the study have the potential to support decision-making?


b. Probe: Any recommended action to HRH2030 to support implementation of the recommendations?

• Do you feel that the HRH2030 support to the Human Resources Management System (HRMS) has the potential to ensure availability of gender segregated HRH data for decision-making use and to improve HRH functions?



Note: All interviews MUST begin with introduction and oral consent, see document #1

KII guide for other USAID partners other than HRH2030 IPs

• Are you familiar with USAID’s HRH2030 project in Jordan?

• Probe: How familiar are you with the USAID HRH2030 project

• Human Resources for Health best practices, as per the USAID and HRH2030 definitions, refer to human resources management and the human resources development best practices. They fall into five main HR areas – capacity, policy and practice, data, performance management, and training and staff development.

For each of these please state if they have improved.

HR areas Better Same Worse HR capacity HR policy and practice HR data Performance management Training and staff development

If there are examples of successes, what are they?

• Among the following practices, which have improved, stayed the same, worsened?

Practices Better Same Worse Recruitment Orientation Retention Motivation Career development Leadership Team building Coaching Support and supervision Communication

Probe: To what do you attribute the changes or lack thereof?

Probe: What if any was the contribution of the HRH2030 to any changes?

Probe: Was there a contribution from your own organization/ activity to any of the above?

Probe: What future recommendations are needed to improve HRH?

Probe: For the HRH2030 project specifically?

Describe any opportunities for your project had to cooperate or coordinate with HRH2030 in the area of training? Were there areas of overlap?

How do you rate the status of the health work force now as compared to two years ago? What changes if any can you identify?


Areas Better Same Worse – Capacity in numbers – Distribution among urban and rural – Working conditions – Motivation – Competence among Managers – Accountability – Transparency – Use of data and knowledge – Workforce retention

Probe: What in your view may be the reasons for changes or the lack of change?

Are you aware of achievements, improvement or progress in the MOH in any of the following areas?

Areas Changes – Leadership training – Policies – Resources – Coordination – Advocacy – Governance

Describe any HRH 2030 activities you are familiar with that seek to improve women’s access to management and leadership positions

a. Probe: Do you feel there was progress in this area in the last two years?

b. Probe: What contributed to such changes?

c. Probe: If there was no or little change, why is that?

What do you know about the new law requiring continuing professional development for Jordan’s health workforce?

a. Probe: What are in your view the necessary actions or elements for its implementation?

b. Probe: Do you have examples of activities that your organization wishes to see done by the MOH in this area?

c. Probe: Does your own organization have plans in this area?


KIIs – WHO and Int’l Orgs


• What do you know about HRH2030?

a. Probe: Is it having an impact on the MOH health sector? Why or why not?

• Are HR practices at the MOH improving? How are they improving?

a. Probe: What are the signs that it is or is not?

b. Probe: What is driving that improvement?

c. Probe: What kind of improvement?

• How is health workforce retention in the MOH sub-sector improving?

a. Probe: What signs do you see of improvement / lack

b. Probe: What is needed to improve it?

• What do you know about HRH2030 support to women in the MOH health sector?

a. Probe: Describe activities if needed

b. Probe: Do you think these are the correct interventions?

c. Probe: What else do women need for success?

• In what ways are women’s opportunities for management and leadership in the health sector increasing?

a. Probe: Why or why not?

• What do you know about the new law on CPD?

a. Probe: (if yes) What do you think it needs to succeed?

• To what extent was your organization involved in developing the National HRH strategy?

a. Probe: (if yes) Describe your involvement

b. Probe: (if not) Why?

• How do you think the National HRH Strategy has addressed adequately gender considerations?

a. Probe: Any recommendations to HRH2030 to support achieving that in the implementation plan?

• What is the role of your organization in supporting HRH capacity building?

• In your opinion, what activities or interventions are still needed to strengthen HRMS and HRD at MOH and national level?


KII Questions – Women and Takamol Project


Questions:

First, I’d like to ask some questions about the experience of women in the health sector:

• What support do women get from HRH 2030?

a. Probe: Is this the support they actually need to succeed in the health sector?

b. Probe: what support do women really need?

• Describe the situation for women in the health sector. How has it changed?

a. Probe: If it is improving, is it improving fast enough? What is improving it?

b. Probe: If it is getting worse, why?

• What are the barriers to women becoming managers or leaders?

a. Probe: What support do women need in overcoming those barriers?

• Describe the attitudes of men in health care facilities about women taking leadership and management positions. Do they support women in taking on these roles?


b. Probe: What can be done to gain the support of men?

c. Probe: What support do women need to take on leadership roles when their male colleagues are not supportive?

• What must be done to retain and increase the number of women in Jordan’s health workforce?

a. Probe: Is anything being done now to support this?

b. Probe: What else needs to be done?

• Are women in your facility involved in top level decision-making?

a. How can you tell they are/are not involved?

Next, let’s talk about the health sector in general:

• Are HR practices at the MOH improving overall, for everyone?

a. Probe: Describe the ways they are improving.

• In what ways is the training received by MOH personnel useful and relevant?

a. Probe: Are people actually able to use that training? Why or why not?

• In what ways is the new law on continuous professional development good for the health sector?


b. Probe: What does it need to succeed?

• What could be done to improve the health sector that is not already being done?


FGD QUESTION GUIDES

Standard FGD Introduction

The facilitator(s) will provide this introduction at every FGD session and ascertain that he/she has the verbal consent of the participants before proceeding.

Introduction:

First of all we would like to thank you very much for your willingness to cooperate in this group discussion. Your participation is voluntary and if anyone feels they do not want to answer any question they are free to do so and free to stop participating at any time. Do we have your consent to participate? Please each one of you say yes if you agree.

The aim of the discussion is to get a better idea about the human resources in the health sector in Jordan and to investigate what the problems and opportunities are for health workers. The discussion today is part of a larger study/evaluation that looks at the last two to three years of activities related to policies and programs in human resources for health in Jordan, and we are talking to various stakeholders.

We would like to have an open and honest discussion with you about different aspects of working in the health sector. We are here to listen to you and learn from you. The discussion will go as follows: The facilitator [Hamouda Hanafi/Khaled Hassan/Raed Azmi/Wisam Qarqash] will lead the discussion. We have prepared a number of topics on which we will ask your opinion.

It is very important that you feel no restraints to speak your mind. Be as open, direct and sincere as you can. The statements you make should be based on real-life experiences and observations. It is important to us to know how the situation really is and not how the situation ought to be, unless we ask you otherwise. Please mention it if you feel you don’t agree with what someone else is saying.

We would like to stress that your personal contributions and views will not be shared with anyone outside this room. To guarantee the anonymity of participation in this discussion, we will not take your name down. We will also have a number of other discussions, so your view will be balanced out with what others say.

Also note that the discussion will be recorded either on paper or on tape or both This is because we do not want to miss anything of what you say. However, nothing of what you say will ever be made public with any of your names attached. The results will only be written down as the report of a discussion with “a selected group of health workers or institutions”. Lastly, the discussion is estimated to last approximately about two hours. You may leave at any time, with no consequences to you.

Are there any questions or remarks?


FGD Questions for health service providers

Facilitator: please read the standard FG introduction and obtain the approval of the respondents to your question about consent before proceeding with the FGD

• What are the most important initiatives taken by your management during the last three years to improve service provider availability, capacity building, performance improvement and retention?

Probe: To what extent you are satisfied with the transparency and fairness level of decisions related to HR functions?

Probe: What additional HRH initiatives do you recommend or wish to see happen?

Which HRH2030 supported capacity building activities did you, your colleagues or your supervisors participate in, and how has it benefitted you/them?

a. Probe: What effect do you think these activities could have?

b. Probe: What action for improvement in these activities do you recommend to HRH2030 Project?

What continuing education opportunities did you have access to during the last three years, and how has it benefitted you?

a. Probe: Has your continuing education been considered by your management in your performance appraisal and in increasing your chance for promotion?

How can HRH2030 support the new law on continuing professional education to ensure the law leads to equitable opportunities for professional and career development?

How effective do you think the current practices are in the area of employment orientation for newly hired or transferred staff at your facility?

a. Probe: Do you think the worksite orientation plans and the presentations for onboarding and general/job-specific orientation, developed with HRH2030 to support, are useful in improving the process?

b. Probe: What actions do you recommend to HRH2030 to support improving these practices and making better use of them in all HDs/facilities?

What do you think needs to be improved in your current job description and performance monitoring tools?

a. Probe: Do you think the use of the competency-based job descriptions and performance management checklists, developed with HRH2030 support, has the potential of improving the capacity and performance of service providers?

b. Probe: Any recommended action to HRH2030 for achieving that?

How do you think the current HRM practices of determining staff positions at the facility level and filling them could be improved?

a. Probe: Do you think the use of the workload indicators of staffing need (WISN) tool, developed with HRH2030 support; will enable developing and implementing a need-based staffing plan at facility level?

b. Probe: What further HRH2030 support is recommended to improve/make better use of it?

Do current HRM practices increase motivation and retention of service providers?


Do current practices support a good environment for women in health care?

a. Probe: Are conditions improving for women in health care?

b. Probe: Do you think the recommendations of the study “the motivation and retention of health workers,” conducted and disseminated with HRH2030 support, have the potential to be implemented in government health services?

c. Probe: What further action do you recommend to HRH2030 to support achieving that?


FGD guide for the Women’s focus group

Facilitators will read the standard FG discussion introduction and obtain the approval of the respondents to the question about consent before proceeding with the discussion.

Discussion topics and questions

First, I’d like to ask some questions about your experience as women:

• What support do you get from HRH 2030, and does it help you to succeed as a woman in the health sector?

a. Probe: what other support do you need?

• Is the situation of women in the health sector improving?

a. Probe: In what ways is it improving?

b. Probe: If it is improving, is it improving fast enough?

c. Probe: if it is getting worse, why?

• Are there barriers to women becoming managers or leaders? What are they?

a. Are the barriers changing?

b. What could be done to reduce those barriers?

• Are men in your facility supportive to women taking on leadership and management roles?

a. How do they show their support? Describe what they do to support or block women in leadership.

b. What else could men do to support women?

c. What needs to be done to increase men’s support to women in leadership roles?

• What is needed to retain women in Jordan’s health workforce?

a. Probe: what could be done to increase the number of women in the health workforce?

• Are women in your facility involved in top decision-making matters?

a. How is their involvement evident?

b. How could women’s involvement be increased?

Next, let’s talk about the health sector in general:

• Are HR practices at the MOH improving overall, for everyone?

a. Describe the ways they are improving or getting worse.

b. What other improvements would you like to see?

• Is the training received by MOH personnel useful and relevant?

a. Probe: Are people actually able to use that training? Why or why not?

• How does the new law on continuous professional development affect the health sector?

b. Probe: Will it strengthen the health sector?


c. Probe: Will it help make the health sector more equitable to health care personnel?

d. Probe: What does the law need to succeed?

• What could be done to improve the health sector that is not already being done?


FGD for HRH staff trained by HRH2030 Activity

The facilitator(s) will read the standard introduction to FGDs and obtain the approval of the respondents to the question about consent before proceeding with the discussion.

Topics and Questions

• Over the last three years, what challenges you have faced in performing your work as HR staff?

• Which HRH functions you think have improved (Planning, Recruitment, Orientation, Deployment, Motivation, Career Development, and Retention)

• What type of training you have received during the last three years? Who were the organizers/supporters?

• In your view, to what extent have the content and methodology of the HR training provided by HRH2030 been effective?

◦ What changes do you recommend to HRH2030 to introduce in the content and methods of training?

• How closely did the participants in the HR training fit the selection criteria for that training?

• To what extent you have been able to apply your new knowledge and skills to improve HR functions?

◦ What have been the enabling factors in applying newly acquired skills?

◦ If there have been barriers, what are they?

• To what extent you think the HRH2030 support to strengthen the HRMIS has enabled the system to collect sex segregated data, analyze them, and making them available for decision making use?

◦ Does the use of this data improve transparency and fairness?

• What guiding policies, regulations and tools do you have to perform your job with clear direction?

◦ What additional policies would make your work easier and more effective?

• To what extent do you think the current posting of HRH staff, capacity building and career development is fulfilling your expectation?

• How could HRH careers be made more fulfilling and attractive to new recruits?

• Do your peers and managers understand and support your HR role?

• Is there anything else you’d like to tell us about HRH in the MOH or the HRH2030 project that we did not discuss?


FGD with Mid-level & Facility Managers

The facilitator(s) will read the standard introduction to FGDs and obtain the approval of the respondents to the question about consent before proceeding with the discussion.

FGD topics and questions

• What activities of the HRH2030 Project have you, your staff or colleagues have participated in?

a. Probe: Which of those activities have the potential to improve provider skill mix, gender balance, performance and retention?

b. Probe: What do you recommend to improve health workforce recruitment development and retention practices?

• What components of the HRH2030 training on management and leadership are relevant to mid-level management competencies?

a. Probe: Which trainings have the potential to improve manager performance and increase women’s participation in leadership roles?

b. Probe: How could HRH2030 improve the training?

• What type of training for HR staff at the HD/facility level has been provided with HRH2030 support?

a. Probe: What additional training should HRH2030 support?

• What was learned from piloting the WISN tool in Balqa?

a. Probe: How can it be made more useful in developing and implementing need-based staffing plans for your HD/facility?

b. Probe: What further HRH2030 support is recommended to make it more useful?

• How do you think the use of the competency-based job descriptions and performance management checklists could be made more useful in improving the capacity and performance of service providers at the HDs/facility level?


• How do you think HRH2030 can support the use of the recently approved health professionals relicensing regulation in order to develop a continuing professional development (CPD) system that ensures equitable opportunities for professional & career development?

• What are the activities to strengthen HRMS that are implemented at your HD and facility levels, with HRH2030 support?

a. Probe: Do they ensure the availability of the needed gender segregated HRH data for decision-making use at the HD level and facility level?

b. Probe: What actions you recommend to HRH2030 to support that?

• What are the lessons learned from the use of the worksite orientation plans and the presentations for onboarding and general/job-specific orientation, by Ramtha and Irbid Health Directorates, for the orientation of newly appointed employees?

a. Probe: What actions you recommend to HRH2030 to support improving them and making better use of them in all HDs/facilities?


QUANTITATIVE DATA COLLECTION TOOLS

590 Jordan HRH Evaluation

Interviewers/Data collectors: Please provide the standard statement (introduction to the survey) before you distribute the survey form and obtain the verbal consent of the respondent before handing him/her the form.

Survey Questionnaire

Section (1):

ID00 Survey No. ID01 Date / /2018 ID02 Facility Name

ID03 Governorate Amman Balqa Irbid

Section (2): General Information

G1 What is your field of work

1. Medicine 2. Dentistry 3. Nursing 4. Pharmacy

5. Midwifery 6. Laboratory 7. Administrative function

(HR, Finance, Admin)

G2 Academic Qualification

1. Diploma 2. BA 3. MA 4. Other

G3 Years of experience at MoH

G4 For the last three years, how many supervisors did you have?

G5 Do you have a supervisory role? 1. yes 2. no (Go to G7)

G6 How many people do you supervise?

G7 How many times in your total employment at MoH have you been promoted?

G8 Do you have a computer at work? 1. yes 2. no

G9 Do you have access to the internet at work? 1. yes 2. no

Please indicate to what extent you agree with the following statements under the available options:

(1) Strongly disagree (2) Disagree (3) Neutral (4) Agree (5) Strongly agree Leave blank if it is not applicable


Section (3): Training

No answer

Strongly agree

Agree Neutral Disagree Strongly disagree

Paragraph Code

I am satisfied with the career advancement opportunities available to me

T1

The MOH is dedicated to my professional development

T2

I am satisfied with the job-related training the MoH offers

T3

I am aware of the healthcare service quality standards

T4

The training I received contributed to better daily communication with my coworkers, supervisors or supervisees

T5

I would like to attend training courses offered by HRH2030

T6

T7 Have you attended any training with the HRH2030 project? If no (Go to question number C1)

1. Yes 2. No

T8 My Supervisor was supportive of my participation in the HRH2030 training

Strongly disagree 2. Disagree 3. Neutral 4. Agree. 5. Strongly agree

T9 The training that I attended with HRH2030 was useful


T10 The training that I attended with HRH2030 was sufficient


T11 In my daily work, I was able to apply and practice what I learned at the training

Strongly disagree 2. Disagree 3. Neutral 4. Agree. 5.Strongly agree

T12 Would you recommend the training to other colleagues

1.Yes 2. No

Section (3): Communication

No answer

Strongly agree


Paragraph Code

My level of knowledge about the new National HR Strategy is sufficient

C1

I am aware of the by law for renewal of health professional license

C2


No answer

Strongly agree


Paragraph Code

I receive adequate guidance about new policies and regulations when they are issued

C3

I have sufficient oppurtunities to ask questions of my supervisors

C4

I received answers to all the questions I pose to my direct supervisor

C5

Communication with my management and coworkers is adequate

C6

There are regular meetings between staff and management

C7

Section (4): Working environment

No answer

Strongly agree


Paragraph Code

I have a copy of my job description

P1

I am thinking about looking for an opportunity outside MoH

P2

I see myself still working at MoH in 5 years’ time

P3

I am satisfied with the level of motivation I receive at the MoH

P4

My working conditions are satisfactory

P5

I receive meaningful recognition for doing good work

P6

I am given opportunities to learn and develop my skills

P7

I Think I have a clear career path in my job

P8

I have an equal chance as others for getting promoted

P9

I believe the recruitment process at MoH is transparent and fair

P10

I am satisfied with the salary and benefits I receive for my job

P11


No answer

Strongly agree


Paragraph Code

My salary and benefits are fair compare to the local market

P12

The MoH recognises good job performance

P13

The new by law for renewal of health professional license is an opportunity to improve medical care

P14

P15 Do you have any other comments or suggestions?

Introduction for respondents to Survey

Interviewer: please read this introduction and obtain the approval of the respondent to your question about consent before proceeding with the interview.

My name is….. and I would like to thank you very much for your willingness to participate in this survey. The survey aims to learn about human resources in the health sector in Jordan and to investigate challenges and opportunities for health workers. This is part of a larger study/evaluation that looks at the last three years of activities related to policies and programs in human resources for health in Jordan, and we are talking to various stakeholders. We are also conducting interviews with many officials and professionals from various sectors and organizations related to the subject of human resources for health.

Your participation is totally voluntary, and you can choose to not answer any question, or to end the survey at any point, with no negative repercussions.

Your personal responses are anonymous, and we ask you to make sure you do not write your name on the form.

The results will only be written down as the report of a survey of “a selected group of health professionals and workers.”

Do I have your consent to reply to the questionnaire?

(Interviewer: if the answer is yes please give the respondent the survey form to fill out)


ANNEX III. DOCUMENTS REVIEWED RESEARCH STUDIES AND ASSESSMENTS

1. Jordan Women in Health Management Study, HRH2030, 2018

2. National HRH Observatory Assessment Final, HRH2030 2017

3. Jordan National Health Sector Strategy 2015-2019

4. Summary of women’s leadership findings, HRH2030 2018

5. Baseline Assessment, HRH2030

6. Motivation and Retention of Health Workers in Ministry of Health Facilities in Four Governorates in Jordan, Findings from a Mixed Methods Study Ministry of Health, HRH2030, 2016

7. Human Resources Systems and Capacity Needs Assessment, HRH2030: Human Resources for Health in 2030, Version 22017 Jordan Gender Strategy, HRH2030 2017

8. National Human Resources for Health Strategy, High Health Council, 2018

9. Human Resources for Health Policy Solutions; HRH2030: Human Resources for Health in 2030

10. Health Finance and Governance Activity Assessment of the Government Finance and Management, HFG 2018

11. Assessment of the Government Finance and Management Information System in the Ministry of Health USAID Health Finance and Governance Activity Decentralization in The Public Healthcare of Jordan – a Situational Analysis 2017, HFG 2018

12. Proposed Client and Community Feedback System; HRH2030, 2016

13. Assessment of Jordan MOH HR policies and procedures, Initiatives Inc., 2005

14. Briefing book on Jordan; USAID Energy II IQC, March 27, 2008

15. Decentralization in the Public Healthcare of Jordan- a situation analysis; USAID HFG Project; June 6, 2017

16. Handbook on M&E of HRH; USAID, World Bank. WHO; 2009

17. HRH indicator compendium; USAID Capacity Plus Project; Intra Health International; June 2011

18. HRH Indicators Reference Sheets; USAID Health Finance and Governance Project; 2015

19. Human resource development plan Jordan; USAID Human Resources Development Project Report No. 8, September 2005

20. Institutional Support and Strengthening Program (ISSP) Jordan End of Project Evaluation; USAID; December 2013

21. Jordan civil society assessment; USAID Monitoring and Evaluation Support Project (MESP), Jordan; May 2016

22. Jordan human resource assessment situation analysis; Jordan USAID, Jordan Ministry of Health, Initiatives Inc.; January 2005


23. Motivation survey Jordan human resource assessment; USAID, CDG engineering and management Associates/ Community Development Group, Jordan, July 2005.

24. Proposed client and community feedback; HRH2030, June 16, 2016

25. Special development objective 4 performance evaluation; Jordan Evaluation Support Program (MESP), 2017

26. WISN Users’ Manual, World Health Organization, 2010

27. Women enrollment in health workforce literature review, HRH2030

AWARDS AND CONTRACTS

• 2015 08 27 Original Award • TOR for embedding MOH draft points v(4) (2)

HRH2030 PERIODIC WORK PLANS AND REPORTS

• Work Plan Year 1 • HRH 2030 Year 2 Work Plan • HRH2030 Jordan Year 3 Work Plan • HRH2030 Y2 Annual Report • HRH2030 Jordan Q3FY18 • HRH2030 Jordan Q2FY18 • HRH2030 Jordan Quarterly Report Q3 2016 • 2016 05 03 Q2 Report Final • HRH2030 Jordan Annual Report • 2016 05 03 Q2 Report • HRH2030 Jordan Q2FY18 Report • HRH2030 Jordan Q1FY18 Report • HRH2030 Jordan Q3FY17 Report • HRH2030 Jordan Q2FY17 Report • HRH2030 Jordan Q1FY17 Report

MOH POLICIES, REGULATIONS, LAWS, PROCEDURES

• MOH operational policy work 9-5-18 • MOH law and Relicensing law in Jordan Gazette • Taskforce meeting 1 minutes • MOH Onboarding and new employee orientation • MOH HRD planning and training plans • MOH HR Planning • MOH Employee Handbook • MOH Developing policies and procedures • MOH Developing competency-based job descriptions • MOH Succession Planning • MOH Policy Training Agenda • Private Hospitals Law • Health Accreditation Law • Health Indicators Policy


• Public Health Law • MOH Structure Regulation 2018

MONITORING AND EVALUATION

• HRH2030 2017 AMELP • HRH2030 2017 PIRS • HRH2030 PMP

WOMEN’S FORUM

• TOR Women Network, HRH2030 • Women in Health Leadership Steering Committee Meeting Minutes

RE-LICENSING AND CPD

• TOR, National Committee for CPD • Re-licensure and CPD Stakeholder Analysis, HRH2030 • CPD research, HRH2030 • Part 3, the CPD model, HRH2030

TECHNICAL GROUPS AND WORKING GROUPS

• Taskforce membership & TOR, Arabic • Taskforce membership & TOR, English • Taskforce meeting 1 • NHRHS Advisory Committee TOR • NHRHS Advisory committee formulation & TOR, Arabic • NHRHS Advisory Committee 2nd meeting notes • NHRHO HHC Focal point training • NHRHO meeting notes, Arabic • NCPF meeting notes, Arabic • NCPF Formulation & TOR, Arabic • List of all HRH2030 Training Participants

The evaluation team also reviewed all training materials included training modules, notes, and assessment materials.


ANNEX IV. ADDITIONAL FEEDBACK ON TRAINING This annex provides additional feedback on the training experience from participants, as well as additional recommendations.

FEEDBACK ABOUT TRAINING IN HML

1. Length of training: Most participants said two years to complete training was too long, and that the same content could have been covered just as efficiently in a shorter period.

2. Problems with selection criteria: Participants were discouraged by lack of adherence to official selection criteria for the training. This also reinforced the perception that wasta governs the health care system.

3. Class size and facilitation: The popularity of the HML course surged as it became known that attendees would be promoted to higher posts. The group increased from 40 to 67 participants; organizers split the group after a few crowded sessions. Each of the original two trainers was assigned to one group. The two trainers were of different level of expertise and training experience. The change also resulted in having two groups meet in two smaller rooms, which participants said were uncomfortable.

4. Trainers: The two trainers had different levels of expertise and training experience, and participants consistently complained about the less experienced trainer’s teaching and facilitation skills. They felt the more experienced trainer explained and talked about each slide in the PowerPoint presentations, while the other was just reading the slides. The group assigned to the less experienced trainer felt they had been shortchanged. However, the more experienced trainer required simultaneous translation, which doubled the time needed and reduced the quality of facilitation.

5. Adult education methods were not applied: The majority of participants felt the exams at the end of the sessions were inappropriate for adult learning. Participants also said that they were not given the correct answers afterwards, so they did not know which answers they had gotten right or wrong. Homework assignments were not reviewed and discussed; the trainer said there were too many participants to make it possible to give feedback to each one.

6. Sharing of evaluation results: Although each session was evaluated and the results were included in HRH2030 reports, participants said they were not shown evaluation results or given feedback about their comments and complaints.

7. Other feedback: Some participants said they could not hear the facilitator’s voice; they could not understand some concepts or models presented; there were inconsistencies in the distribution of course materials; participants perceived as being “more senior” received preferential treatment from the facilitators; and lack of adherence to time on the part of participants.

RECOMMENDATIONS SPECIFIC TO HRH2030 TRAINING

8. Senior MOH supervisors and managers should be invited a few at a time to observe training sessions so they can familiarize themselves with the content and show their support for the trainees.

9. Copies of schedules, agendas, and training materials should be shared with the trainees’ immediate supervisors.


10. Training reports and materials should be shared with senior MOH staff.

11. Trainers should provide feedback to the trainees about their homework, daily evaluations, and final training evaluation.

12. Facilitation should be customized to address differences in participants’ skills level. Facilitators should have the same skill level, especially if similar workshops are being run simultaneously.

13. Participants’ feedback about issues with training should be addressed, and participants should receive responses to their feedback.

14. Exams based on rote memorization should be replaced by methods suitable for adult learning. After tests are scored, the correct answers should be provided and there should be a class discussion about the questions.


ANNEX V. STATEMENT OF DIFFERENCES The Human Resources for Health in 2030 (HRH2030) Activity welcomes the opportunity to reflect on its performance to date, to identify areas of success at the midpoint of activity implementation, and to assess areas for improvement. We are grateful to USAID/Jordan for initiating a midterm evaluation of HRH2030/Jordan’s activities, which will further inform future health workforce interventions in Jordan. However, we believe that several of the methodologies and data collection tools used during the midterm evaluation were inappropriate or incorrectly applied, and, therefore, several of the conclusions drawn from the evaluation are incomplete or reflect aspects of the health system outside of the scope of HRH2030.

Below, HRH2030 elaborates on its concerns with the methodology used to complete the midterm evaluation and the conclusions outlined in the midterm evaluation report. We provide specific examples illustrating our concerns.

STATEMENT OF DIFFERENCE ON MIDTERM EVALUATION METHODOLOGY

• Incorrect Application of Performance Evaluation Approach

As set forth in USAID’s Automated Directive Systems (ADS) chapter 201, performance evaluations during the course of an activity must conform to either impact or performance evaluation approaches. While evaluators are granted wide latitude in designing their approaches, ADS 201.3.5.12 notes that “the selection of method or methods for a particular evaluation should principally consider the appropriateness of the evaluation design for answering the evaluation questions as well as balance cost, feasibility, and the level of rigor needed to inform specific decisions.”

In reviewing the methodology used to gather, evaluate, and draw conclusions for this midterm evaluation, HRH2030 does not believe that the selected methods of evaluation meet USAID’s standards for a performance evaluation. The evaluation report notes that:

“While the project’s outputs were available through the periodic reports, the data collection tools did not seek to measure the Activity’s efforts and inputs, or the speed and cost-effectiveness of its execution, and there was no direct intent to measure levels of achievements against pre-stated midterm objectives. Although some of the evaluation questions speak of changes in the MOH human resources (HR) situation and application of training results, this evaluation was not expected to measure changes in the indicators identified in the HRH2030 Activity Monitoring, Evaluation, and Learning Plan (AMELP), but primarily aims to provide qualitative information and insights to respond to the evaluation questions…” [page 1].

Ignoring the Activity’s inputs – such cost or feasibility of activities – as well as the design of HRH2030’s activities against its AMELP is a methodology more appropriate for a formative evaluation, where less rigorous methods are used to measure progress in order to guide implementation. Instead, the evaluation draws authoritative conclusions on HRH2030’s performance despite being explicitly designed as a formative evaluation [page 32] and using less reliable evaluation methods that lack the rigor required of performance evaluations. Conflating two distinct evaluation approaches—formative and performance evaluation—and implying that the former can achieve the latter, is contrary to established evaluation standards, including USAID’s own as specified in its ADS.

Moreover, the evaluation report itself notes, “the evaluation questions posed by USAID look at changes to the health system as a whole rather than specific effects of HRH2030” [page vi]. However, rather than caveating the evaluation based on that limitation, the evaluation appears to set its own expectations of changes in specific outcomes tied to answering the questions that are not aligned with the


performance benchmarks set for HRH2030 in the annual implementation plans or the AMELP approved by USAID/Jordan. Specifically, the expected changes that underpin the evaluation are broad and long-term in nature – such as changes of organizational culture and individual behavior rooted in culture and tradition – and should therefore be beyond the scope of a mid-term evaluation; changes in development outcomes at the level of the evaluation would be more appropriate for an impact evaluation.

• Inappropriate Design of Data Collection Instruments

HRH2030 believes that several of the data collection instruments used in the evaluation are inappropriate or incorrectly applied. For example, the evaluation used a quantitative survey to assess progress against outcomes [pages 3-4]. However, the design of this quantitative survey [pages 66-69] did not meet the rigor of a USAID midterm performance evaluation.

First, its questions were intended to measure participants’ opinions about their role and satisfaction within the Ministry of Health system – which goes far beyond the objectives of the HRH2030 Activity – rather than about the HR development/management (HRD/HRM) trainings provided by HRH2030. Moreover, by specifying HR at the Ministry of Health in general as the target for this survey, it is unlikely that interviewees would associate their responses with specific activities with the HRH2030 Activity; instead, it is much more likely that interviewees interpret it as a general reference to human resources for health (HRH) or to WHO’s Global Strategy on Human Resources for Health: Workforce 2030, which would make it impossible to assess the HRH2030 Activity’s impact on these broad long-term outcomes, despite the evaluation report’s claims.

Second, the design of this quantitative survey encourages biased responses by being non-specific; for example, it asked if training was “useful” or “sufficient” without specifying the training objectives [page 72, questions T9 and T10] and instructed interviewers to “Leave blank if it is not applicable” [page 66]; standard rigorous evaluation practice is to relate questions to training objectives and include ‘N/A’ as a response option. Also, the quantitative survey instrument uses a 5-point Likert response scale including ‘Neutral’ as the middle category [pages 66-69]. Typically, most responses in this scale fall into the “Neutral” category leaving few responses that provide information for a meaningful analysis. Whether or not this instrument provided any useful information is impossible to tell because complete results from the data analysis were not presented.

Third, the focus group discussion (FGD) guides developed as part of the mid-term evaluation are also similarly flawed. There is no effort to distinguish questions about broader opinions about systemic HRH issues from the specific areas that the HRH2030 Activity was designed to address. Furthermore, several questions in the discussion guides are negatively leading by implying that there are existing failures or shortcomings, encouraging respondents to suggest “improvements” [pages 60-65].

Additional examples of the flaws in the evaluation’s design include statements that draw conclusions without appropriately considering the limitations of the qualitative and quantitative data collected, including “It is possible that participants felt that the training was useful and simultaneously also that it was flawed,” when survey data appear to contradict FGD findings [page 11], and “To illustrate this, the quantitative survey showed that only 18 percent of those who were trained, and 11 percent of the others feel that MOH recognizes good job performance. A somewhat larger proportion of respondents said they are getting meaningful recognition for doing good work — 58 percent of those trained by HRH2030 and 45 percent of those not trained.” [page 12].


STATEMENT OF DIFFERENCE ON MIDTERM EVALUATION CONCLUSIONS

• Incomplete or Unqualified Conclusions

HRH2030 believes that the conclusions drawn in the evaluation are incomplete or show a lack of knowledge and understanding of the structures and actors in Jordan’s health sector. For example, the evaluation publishes sample feedback from training in Annex IV [pages73-74] of the evaluation report but does not provide context about how participants scored training sessions or the context in which these statements appeared; without a more thorough publication of questionnaire results, published comments could easily be a small minority opinion (as little as one respondent) that does not provide readers of the evaluation report with a full picture of participants’ opinions of HRH2030’s interventions.

In addition, even where the evaluation report states serious limitations to the evaluation’s methodology, it does not qualify the findings and statements appropriately. For example, while the evaluation report notes that interviewees were not selected randomly, it does not specify how many of the 259 interviewees were training participants and how many were not exposed to any HRH2030 activities [page 4]. Despite this, the evaluation claims on page 4 that “Triangulation across data sources minimizes the effect of these limitations.” There is no evidence that data triangulation was used in this evaluation which relies exclusively on opinions and perceptions; triangulation without additional unbiased data sources cannot correct for inaccurate and biased information.

• Incorrect or Contradictory Statements

Statements in the evaluation report are exclusively based on opinions and perceptions, which poorly represent actual practices and behaviors and ignore evidence from published research including the HRH motivation and retention study conducted by HRH2030, which was available to evaluators. The propensity of interviewees to convey negative opinions about low salaries and poor working conditions [page 10] needs to be understood in a context where the MOH sector offers many desirable conditions and has a very low employee turnover rate. Similarly, opinions such as “Fewer than half of the service providers surveyed saw themselves still working at MOH in five years’ time” [page 11] are not taken in context with studies such as the HRH2030 motivation and retention study, which shows high healthcare worker retention in the Ministry of Health.

The conclusions also seem to be drawn exclusively on negative opinions, ignoring any positive feedback; for example, data displaying the considerable differences in satisfaction between those trained by HRH2030 and those not trained by the Activity. Examples can be found throughout the report, including findings such as “Only 16 percent of the staff surveyed indicated they are satisfied with the job-related training that MOH offers compared to 52 percent satisfaction among those surveyed who participated in HRH 2030 training” [page 12].

Importantly, several of the statements and conclusions in the evaluation report contain incorrect assertions. For example, the evaluation confuses continuing professional development (CPD) requirements with bylaws for relicensure [page 9, the author inaccurately added (continuing professional development) to the quote “HRH2030 prepared the (continuing professional development) bylaw…”]. Although the report states that “The terms ‘relicensing’ and ‘CPD’ have been used in the same context and interchangeably in our discussions with key informants and focus groups” [page 14], this is not an appropriate interchange.

The evaluation makes incorrect conclusions about HRH2030’s approach to sustainability in relation to the activity’s agreement with USAID/Jordan. The evaluation report states that the project lacks a sustainability plan and recommends the development of one [page 24] and that “at present, each project intervention has a sustainability component, but there is no overall project sustainability plan” [page 24]. HRH2030’s agreed approach with USAID/Jordan was to have sustainability built into each activity in the


annual implementation plan, which was done in the milestones table at end of each intervention, the Five-Year Vision in the annex of the Year 2, Year 3, and Year 4 implementation plans, and the approved AMELP.

Lastly, many of the recommendations are not based on the evidence collected during the evaluation, which is a requirement per USAID guidance. Statements such as “…there is some doubt as to whether WISN [Workload Indicators of Staffing Need] data will actually be used…” [page 25] are pure speculation. Similarly, the recommendation of weekly instead of monthly training [page 26] contradicts interviewee responses about staff’s time constraints.

CONCLUSION

We believe that the midterm evaluation of HRH2030 in Jordan was based on evaluation methodologies and data collection tools that were inappropriate or incorrectly applied. The evaluation report makes unqualified or factually incorrect statements and emphasizes several issues which are beyond HRH2030’s scope of work or manageable control. This evaluation report reflects a limited understanding of the HRH2030 Activity, its key accomplishments, and its performance against HRH2030’s approved AMELP. As a result, we respectfully disagree with many of the findings and conclusions included in this midterm evaluation report.

HRH2030 appreciates the efforts of the evaluation team and of USAID/Jordan in initiating and completing this midterm evaluation. We hope this feedback is useful and can be incorporated into this and future efforts. We are happy to provide additional information related to the outlined concerns or overall performance of HRH2030, as requested.


ANNEX VI. DISCLOSURE OF ANY CONFLICTS OF INTEREST








ANNEX VII. SUMMARY BIOS OF EVALUATION TEAM Alanna Shaikh, Team Lead, has 20 years of experience in global health, primarily in USAID-funded health systems strengthening programs. For this evaluation, she conducted qualitative data collection and analysis and provided overall technical and logistical guidance. She has worked in the Middle East, Central Asia, and West Africa, including as regional director for monitoring and evaluation on a four-country health system strengthening program and as health and education officer director for USAID/Kyrgyz Republic. She has a master’s in public health from Boston University and an undergraduate degree in Middle East studies from Georgetown University.

Raed Azmi, Evaluation Expert, has more than 10 years of evaluation experience in large, complex, and multilateral funded projects on a national, regional and international platforms. He has designed and conducted studies and research methodologies, including stratified random sampling, and developed tools for focus groups, surveys, and key informant interviews. He has also conducted quantitative and qualitative analysis and modeled statistical data with SPSS and Stata. For this evaluation, he provided technical oversight, quantitative data collection and design, and data analysis. He has a Master’s in economics from Birzeit University in Palestine.

Hamouda Hanafi, International HRH Expert, has 25 years of experience in senior executive positions. He provided qualitative data collection, design and analysis on this evaluation. His field experience includes the Middle East, North Africa, and West Africa, particularly in Tunisia, Egypt, Yemen, Niger, and Cote d’Ivoire. His technical areas of expertise include behavioral change communication, training development and facilitation, developing and conducting capacity assessments, and evaluating programs and performance. He has a Ph.D. in public health from the University of California at Berkeley.

Khaled Hasan, Jordan HRH Expert, is a physician with more than 20 years of experience in health care quality management and HRH with the Ministry of Health of Jordan, the Health Care Accreditation Council of Jordan, and WHO. For this evaluation, he provided qualitative data collection and design and wrote technical content. His technical skills include conducting surveys and operational research, developing policies and strategic plans, and assessing health professionals’ education/training clinical competencies. He has an M.Sc. in hospitals and health facilities management from the University of Jordan and has degrees in medicine and surgery from Mosul University in Iraq.

Wisam Qarqash, Field Coordinator, has more than 20 years of experience designing and implementing health and nutrition programs in Jordan. For this evaluation, she provided logistics, qualitative data collection, and support. Her experience includes work with the Ministry of Health, the Johns Hopkins Bloomberg School of Public Health, Center for Communication Programs, and WHO. She has a degree in nutrition from Jordan University.

For more information, please visit


Global Health Program Cycle Improvement Project 1331 Pennsylvania Avenue NW, Suite 300

Washington, DC 20006 Phone: (202) 625-9444

Fax: (202) 517-9181 http://ghpro.dexisonline.com/reports-publications

Date post:	30-Jan-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times