New MID-TERM PERFORMANCE EVALUATION · 2018. 12. 12. · Inc. MID-TERM PERFORMANCE EVALUATION...

August 9, 2013

This publication was produced at the request of the United States Department of Agriculture, Foreign Agricultural Service. It was prepared by Andrew Gilmour, James Hanson, Syed Kamal, and Mohammed Ajmal though Social Impact Inc.

MID-TERM PERFORMANCE EVALUATION EVALUATION REPORT AFGHANISTAN AGRICULTURAL SANITARY AND PHYTOSANITARY PROGRAM

Mid-Term Performance Evaluation Evaluation Report Afghanistan Agricultural Sanitary and Phytosanitary Program

Conducted by Social Impact 2300 Clarendon Boulevard Suite 1000 Arlington, VA 22201 USA Cover page photograph: CVDRL laboratory, MAIL (site of program workshops) DISCLAIMER: The author’s views expressed in this publication do not necessarily reflect the views of the United States Department of Agriculture or the United States Government.

CONTENTS Acronyms ........................................................................................................................................................... 1

Executive Summary .......................................................................................................................................... 2

Program Background, Purpose, & Evaluation Questions ......................................................................... 7

Evaluation Methods & Limitations ..............................................................................................................10

Findings .............................................................................................................................................................13

Conclusions......................................................................................................................................................30

Recommendations ..........................................................................................................................................32

Annex I: Evaluation Statement of Work ...................................................................................................37

Annex II: Key Documents Reviewed .........................................................................................................39

Annex III: Program Expenditure Rate Analysis ........................................................................................41

Annex IV: List of Key Persons Interviewed ..............................................................................................42

Annex V: Data Collection Instruments .....................................................................................................44

ACRONYMS ACD Afghan Customs Department

AHDP Animal Health Development Program

CAREC Central Asia Regional Economic Cooperation

CEO Chief Executive Officer

CVDRL Central Veterinary Diagnostic and Research Laboratory

DAIL Directorates of Agriculture, Irrigation, and Livestock

EPAA Export Promotion Agency of Afghanistan

GIRoA Government of the Islamic Republic of Afghanistan

HACCP Hazard Analysis Critical Control Points

IPPC International Plant Protection Convention

MAIL Ministry of Agriculture, Irrigation and Livestock

MoPH Ministry of Public Health

NHLP National Horticulture and Live Stock Program

SPS Sanitary/Phytosanitary

USDA/FAS United States Department of Agriculture Foreign Agricultural Service

Mid-term Evaluation Afghanistan Agricultural Sanitary and Phytosanitary Program (AASPS)

EXECUTIVE SUMMARY DEVELOPMENT PROBLEM & PROGRAM BACKGROUND Agriculture accounts for one half of the Afghan economy and employs 75 percent of rural workers.1 However, the lack of food safety and the absence of sanitary/phytosanitary (SPS) systems jeopardize public health and restrict exports.

The Afghanistan Agricultural Sanitary and Phytosanitary Program (SPS Program), funded by the U.S. Department of Agriculture‘s Foreign Agricultural Service (USDA/FAS) and implemented by Purdue University, aims to support the Government of Afghanistan’s Ministry of Agriculture, Irrigation and Livestock (MAIL) in building SPS capacity nationally. According to the terms of the SPS Program’s cooperative agreement, the program’s dates of implementation are from September 21, 2011 to September 30, 2014 with a total budget of $2,175,242. Increased SPS capacity within the Ministry would allow for broader implementation of food safety methods, certification of agricultural exports as pest- and disease-free, and capacity to inspect and test imports of foods and other produce that might be vectors of plant disease, insect pests, or invasive species more broadly defined.

To accomplish its strategic program goals, Purdue divided the SPS Program into three main objectives. As stated in Purdue’s Expression of Interest,2 these objectives include:

1. Build capacity to implement regulatory systems for quarantine and surveillance, incorporating international standards for plant health, food safety, and animal health into Afghanistan’s laws and regulations;

2. Improve laboratory diagnostic capabilities for plant health, food safety, and animal health; and

3. Train Ministry of Agriculture, Irrigation and Livestock (MAIL) and the Directorates of Agriculture, Irrigation, and Livestock (DAIL) staff in risk management for plant health, food safety, and animal health.

PURPOSE OF THE EVALUATION The purpose of this mid-term evaluation is to identify mid-term program results, examine how they relate to program goals and objectives, and determine the likelihood of program sustainability. The evaluation will also serve as a tool to help guide any mid-program changes needed to realign activities to overall goals and objectives.

EVALUATION METHODS This evaluation employed a standard, mixed-methods approach applying the following data collection methods: desk review, key informant interviews, and a trainee survey. With approval from USDA, the evaluation team analyzed program activities from the SPS Program’s inception on September 9, 2011, through December 31, 2012. The evaluation field component occurred from June 10 to June 26, 2013. 1 World Bank, Afghanistan: Priorities for Agriculture and Rural Development, from: http://go.worldbank.org/0NRZAEWOT0 2 Contained in cooperative agreement signed September 9, 2011.


FINDINGS The program delivered the vast majority of its activities as a series of workshops. During the field-work component of the evaluation, the SPS Program held five workshops (four in Kabul and one in Herat) and trained 103 people, primarily drawn from MAIL. A total of 27 days of instruction took place in Afghanistan: 24 in Kabul and three in Herat. During the time span examined by the evaluation, the program did not send Afghan professionals for advanced training in the United States (U.S.) as originally planned, but sent one program beneficiary to the U.S. for training.3

Evaluation Question One Is the SPS Program achieving its objectives and to what extent are program indicators being met?

The evaluation team cannot directly answer this question as the listed activities in the SPS Program work plan did not provide enough indicators to accurately track targets. For instance, workshops did not usually specify dates or the number of participants envisioned for each planned activity.

The evaluation team finds that to date, the program devoted nearly all of its efforts toward Objective 2: Improve laboratory diagnostic capabilities for plant health, food safety, and animal health. The program conducted five workshops: two on plant health, two on food safety, and one on animal health in Afghanistan. Key informants and trainees surveyed generally rated the training highly.

The evaluation team found the SPS Program made no measurable progress in the implementation of the activities associated with Objective 1 (Build capacity of middle/upper level MAIL and DAILs to implement regulatory systems for quarantine and surveillance incorporating international standards for plant health, food safety, and animal health into Afghanistan’s laws and regulations) or Objective 3 (Train middle/upper level MAIL and the DAILs staff in risk management for plant health, food safety, and animal health).

The SPS Program was unable to place Afghan students, as planned, at Purdue University in the U.S. despite best efforts to overcome ever-restrictive visa policies.

Evaluation Question Two Is the SPS Program reaching targeted beneficiaries? If not, why not? If yes, what are the results?

The evaluation team did not find that the SPS Program implemented activities or targeted beneficiaries in Objective 1, which supports the building of a regulatory SPS mechanism. Instead, the evaluation team found that the SPS Program focused all of its efforts on building laboratory diagnostic capacity (Objective 2). The SPS Program effectively targets beneficiaries in this area as compared to the terms laid out in the work plan. The evaluation team did not find evidence that the SPS Program implemented activities or targeted beneficiaries in Objective 3, which promotes the establishment of a risk management system.

Evaluation Question Three What internal and/or external factors are positively or negatively affecting implementation? How can these factors be further developed or overcome, respectively?

Based on key informant interviews with MAIL staff, the evaluation team found that the majority of actors directly involved are motivated to participate in the SPS Program and that program instructors are 3 Outside the evaluation time frame, from January 1 to June 25, 2013, the SPS Program has: 1) conducted a four-day Poultry Workshop in Kabul, training three students in February 2013 and 2) sent one Afghan professional for one month’s advanced training at Purdue University in June 2013.


qualified to teach the program’s curriculum. Reponses to the surveys distributed by the evaluation team indicate satisfaction with the course content and instructors.

Through key informant interviews with MAIL and Ministry of Public Health (MoPH) staff, the evaluation team found that the lack of equipment and supplies for laboratories located at MAIL and MoPH is a critically-negative factor hindering the overall success of the SPS Program. An additional negative factor is the fact that the instruction and materials supplied to students is in English, rather than the local languages of Dari and Pashto. This was surprising as local translation in Kabul is now readily available and affordable.

Evaluation Question Four How is the sustainability of the SPS Program being addressed? Are the beneficiaries being followed up with in order to track their progress?

The evaluation team defines sustainability in this program as the establishment of permanent capacity within MAIL to implement and manage SPS and Food Safety training, systems, and regulation. In order to promote sustainability, the SPS Program must achieve all three inter-related and complementary objectives. Given the SPS Program’s uneven progress across its objectives, the evaluation team finds that the SPS Program does not operate sustainably.

Evaluation Question Five The SPS Program was established to train staff via training-of-trainers and direct methods. What evidence is there that this is having a sustained effect?

The evaluation team found that in the five workshops delivered during the evaluation timeframe, there was no formal training-of-trainers component. Nevertheless, several of the graduates of the Plant Diagnosis workshops subsequently taught related material in Herat and Jalalabad.

Evaluation Question Six How has the SPS Program contributed to the creation and implementation of a functioning SPS system and what needs to happen to improve/expand that benefit?

The evaluation team did not find much evidence that the SPS Program engaged with senior MAIL management – at the Director General level and above – to support the establishment of an institutional framework to manage and promote a SPS mechanism.

Evaluation Question Seven How are the implementers working to identify participants? Are they in constant contact with the Ministry?

Purdue worked closely with MAIL counterparts to identify and select trainees based on mutually agreed criteria. Once the two entities agree upon the criteria, then MAIL makes the actual selection of participants. If the program and MAIL representatives have conflicting opinions, they jointly determine topics for the trainings.

Additional Findings Field Program Management

The program lacks a Kabul-based officer to coordinate, manage, and arrange the vast majority of activities from the U.S. This off-site management – rare for a capacity building program of this scale in Afghanistan – presents program challenges, particularly in the coordination of event logistics, communication between program representatives and USDA counterparts, and engagement with MAIL and other stakeholders.


GENERAL CONCLUSIONS

Program training of excellent quality, but training is too short The evaluation team concludes that the training delivered to date is of excellent quality and directly on target to program and student need. However, the duration of course instruction, between two and five days, has been too short, leaving insufficient time for student confirmation and review. Furthermore, the dearth of training supplies and materials – coupled with an absence of course material translated into a local language – render the student learning experience needlessly challenging.

Program has made progress, despite delays against activities laid out in the work plan Afghanistan is among the most challenging countries to build SPS capacity. It is understandable, if not the norm, that development programs start slowly as staff cope with ever-changing conditions and constraints. As of December 31, 2012, 15 months into the 36 month contract, the SPS Program delivered merely five workshops over a total 27 days of instruction Afghanistan. The SPS Program is short of meeting its work plan activities, as reported in Table 1.

Program management structure presents challenges The present, off-site management model, in which the program is run from the U.S. offices at Purdue University, is inefficient and jeopardizes program effectiveness. The intensive nature of this program, characterized by direct training and mentoring, cannot be delivered or managed at a distance. It must be managed as a direct implementation model where key decisions are made in the field.

Implementation rate has been slower than expected Should Purdue maintain its current implementation rate of activities, it will likely only expend half of its $2.175m budget by program end in September 30, 2014.4 To date, Purdue primarily relies on in-house professors to deliver instruction and mentoring; this may contribute to the slow pace of program delivery.

Sustainability is not being sufficiently addressed The SPS Program does not systematically and effectively engage middle and senior level managers at MAIL to help educate, mentor, and support the establishment of an operational SPS mechanism in the Ministry. Unless management engages in the process, supports SPS establishment, and ensures it has the resources to continue, assured sustainability is extremely unlikely.

Program of use, but will not reach strategic intent unless activities address all objectives To date, Purdue almost exclusively focused on developing laboratory diagnostic skills (Objective 2), and made very little progress in the other objectives of building regulatory capacity and risk management systems (Objectives 1 and 3, respectively).

OVERALL RECOMMENDATIONS Although the evaluation only examined data through the end of December 2012, the evaluation team is confident that its conclusions and recommendations remain valid despite recent SPS Program developments.

4 Based on an extrapolation of its expenditures to date compared to its cooperative agreement budget. See Annex III.


1. Review and revise program work plan and implementation schedule Program management, in consultation with Purdue and MAIL, should review the original USDA/Purdue cooperative agreement and current work plan to ensure it directly addresses all objectives and considerably accelerates the number and frequency of workshops.

2. Revise Training Approach The evaluation team recommends that the SPS Program:

• Provide sufficient supplies and basic equipment to facilitate training;

• Establish longer courses/more time for review;

• Make instructor time on the ground productive by ensuring instructors have all support arranged in advance;

• Instructor trips should be two full weeks in duration;

• Provide more frequent training;

• Translate essential course material into local language; and

• Deliver the vast majority of training, capacity building, and mentoring in Afghanistan.

3. Open recruitment to all qualified professionals, with an emphasis on local staff, when possible

4. Establish a Kabul-based Program Officer

5. Explore viability of assisting Afghanistan to meet International Plant Protection Convention

6. Develop MAIL capacity to implement HACCP within the SPS mechanism

7


PROGRAM BACKGROUND, PURPOSE, & EVALUATION QUESTIONS DEVELOPMENT PROBLEM & PROGRAM BACKGROUND Agriculture accounts for nearly one half of the Afghan economy and employs over 75 percent of rural workers.5 Decades of conflict have overshadowed Afghanistan’s strong global reputation for exported goods, such as almonds, pomegranates, pistachios, raisins, and apricots.6 The lack of food safety and the absence of sanitary/phytosanitary (SPS) systems jeopardize public health and restrict exports. As a result of the absence of a functioning SPS system, Afghanistan’s agricultural exports are largely limited to dried fruit and nuts.7 Many farmers cannot maintain clean produce and traders are unaware of food safety regulations and certifications required for export, or how to obtain such permits.8 Moreover, the government lacks the necessary capacity to enact and enforce SPS regulations and has almost no coordination with other countries on safe food standards.9 USDA sees this is a serious impediment to trade and food safety. This view is reflected by U.S. Ambassador Karl W. Einkenberry, who stated:

“Conformity to international SPS standards is a critical part of expanding Afghanistan's economic power. Increased capacity to develop and to implement SPS standards means increased access to international agricultural markets.”10

Improvement in SPS systems is among the criteria for World Trade Organization accession, which Afghanistan plans to attain in 2013.11

The Afghanistan Agricultural Sanitary and Phytosanitary Program (SPS Program), funded by an interagency agreement between the United States Agency for International Development (USAID) and the U.S. Department of Agriculture‘s Foreign Agricultural Service (USDA/FAS), and implemented by Purdue University, aims to support the Afghanistan Government’s Ministry of Agriculture Irrigation and Livestock (MAIL) in building SPS capacity nationally, including widespread implementation of food safety methods, certification of agricultural exports as pest- and disease-free, and reliable inspection of foods and other produce that might be vectors of plant disease, insect pests, or invasive species more broadly defined. According to the terms of the cooperative agreement, the SPS Program’s dates of implementation are from September 21, 2011 to September 30, 2014, with a total budget of $2,175,242.

Theory of Intervention USDA/FAS structured the Afghanistan Agricultural Sanitary Phytosanitary Systems Program (or SPS Program) as a capacity building initiative targeting MAIL and consistent with the objectives of other 5 World Bank, Afghanistan: Priorities for Agriculture and Rural Development, from: http://go.worldbank.org/0NRZAEWOT0 6 USAID Afghanistan, Agriculture, from: http://afghanistan.usaid.gov/en/programs/agriculture 7 Noted in SPS Program Performance Monitoring Plan, p. 3 8 As frequently noted by MAIL, Minstry of Public Health (MoPH), exporters, and Team Leader via direct, extra-evaluation, and observation. 9 MAIL presentation to Central Asia Regional Economic Cooperation (CAREC) Program Workshop on SPS Measures, Bangkok, Thailand, July 26, 2012. 10 US Ambassador Karl W. Eikenberry remarks at the SPS Capacity Building Workshop, April 4, 2011. 11 World Trade Organization, July 2013

8


programs FAS supports in Afghanistan. Program staff did not articulate a theory of intervention at the design phase. As such, the evaluation team constructed one to help guide the evaluation. It should be noted that the funding solicitation did not require that applicants explain a theory of change as part of the proposal.

Program Objectives To accomplish its strategic goals for the program, Purdue divided the SPS Program into three main objectives. As stated in Purdue’s Expression of Interest,12 these objectives are:



3. Train MAIL and the Directorates of Agriculture, Irrigation, and Livestock (DAILs) staff in risk management for plant health, food safety, and animal health.

PURPOSE OF THE EVALUATION USDA commissioned Social Impact to conduct a mid-term performance evaluation to assess the effectiveness of FAS’ SPS Program in Afghanistan. The purpose of this evaluation is to identify mid-program results, examine how they relate to its goals and objectives, and determine the likelihood of program sustainability. The evaluation will also serve as a tool to help guide any mid-program changes needed to realign activities to overall goals and objectives.

Evaluation Time Frame This mid-term evaluation assessed activities implemented from the SPS Program’s inception until December 31, 2012. The evaluation team recommended this cut-off date because it was roughly at the mid-way point of the program and also because Purdue University only supplied formal quarterly activity up to this date. The SPS Program Managers at FAS approved this cut-off date in advance of the fieldwork. Although the SPS Program continued implementation from January 1, 2013, to mid-June (date

12 Contained in cooperative agreement signed September 9, 2011.

Theory of Intervention: Build technical and managerial capacity within MAIL (through program)

To Enable, MAIL to institutionalize SPS practice

To Establish effective SPS/Food Safety Regulatory Mechanism

To Protect domestic foods & Encourage Exports

9


of the field component of the evaluation), activities were limited; during this six month period, the program trained three scientists in the animal science division in Kabul and dispatched one Afghan MAIL specialist to Purdue for intensive training.

EVALUATION QUESTIONS Through its Statement of Work to Social Impact (Annex I), USDA posed the following questions:

1. Is the SPS Program achieving its specific objectives and to what extent are program indicators being met?

2. Is the SPS Program reaching targeted beneficiaries? If not, why not? If yes, what are the results?

3. What internal and/or external factors are positively or negatively affecting implementation? How can these factors be further developed or overcome?

4. How is the sustainability of the SPS Program being addressed? Are the beneficiaries being followed up with in order to track their progress?

5. The program was established to train staff via training-of-trainers and direct methods. What evidence is there that this is having a sustained effect? How can it be improved?

6. How has the SPS Program contributed to the creation and implementation of a functioning SPS system and what needs to happen to improve or expand that benefit?

7. How are the implementers working to identify participants? Are they in constant contact with the Ministry?

10


EVALUATION METHODS & LIMITATIONS EVALUATION TEAM AND DATA COLLECTION The evaluation team was composed of external evaluators, Drew Gilmour (Team Leader) and Jim Hanson, Ph.D., and Afghan evaluators, Syed Kamal and Ajmal Mohammad. John Dennis, Ph.D., provided off-site support and backup. This evaluation employed the following data collection methods: desk review, key informant interviews, and a trainee survey. Each of these methods is described in greater detail below.

Desk Review Prior to field deployment in Kabul, the evaluation team conducted qualitative analysis of various background documents, including the program design document, work plans, periodic program reports (quarterly and training specific progress reports), and other miscellaneous documents. Once in Kabul, the evaluation team collected and reviewed other literature, primarily provided by MAIL and other in-country SPS actors (see Annex II: Key Documents Reviewed).

Key Informant Interviews – Performance Questions The field component of the mid-term evaluation occurred from June 10 to June 26, 2013, in Kabul, Afghanistan. While in the field, the evaluation team conducted 35 key informant interviews. The evaluators developed and utilized an interview protocol13 that included questions about implementation, reach, effectiveness, and sustainability of the program. The team drafted these questions in English and subsequently translated them into Pashto and Dari. The team conducted approximately 70 percent of the interviews in English. The evaluation team worked through interpreters to conduct the remainder in Dari (about 22 percent) and Pashto (8 percent), using hard copy translations as a guide. In practice, these base questions often served as a springboard for deeper thematic exploration.

The evaluation team verified interview responses by asking the same questions or cross-checking during follow-up interviews. For instance, when first interviewed, one senior MAIL manager was polite and reflected positively about the program; however, through subsequent follow up discussions, the same manager became more specific and critical about program activities and management.

Participants in these interviews included:

• MAIL Deputy Minister

• Plant Protection and Quarantine Department

• Animal Health Department

• Academia

• Ministry of Public Health (MoPH)

• Afghanistan Customs Department (ACD)

13 See Annex V – Data Collection instruments.

11


• Ministry of Commerce (Export Promotion Agency of Afghanistan)

• SPS sister programs such as National Horticulture and Live Stock Program (NHLP)

• Plant Biotechnology Laboratory of the Agha Khan Foundation

• Purdue University (the SPS Program implementers)

• USDA Officials

• Some workshop participants

Appendix V contains soft copies of all interviews, surveys, and researcher notes. The key list of documents is included in Annex II.

Trainee Survey The evaluation team prepared a brief survey to be administered to all 107 workshop participants from program start until June 30, 2013. The survey contained 20 questions related to trainee satisfaction, training quality and utility, and recommendations for the future.14

The evaluation team distributed the survey in English, Dari, and Pashto via email to all 107 program participants. In total, 14 responded.15 Following the email round, the evaluation team called the remaining 93 trainees that did not respond to the email solicitation. The evaluation team connected with 24 additional trainees who subsequently responded to the survey over the phone. In total, the team obtained data from 34 out of the 107 trainees. The team does not have data on reasons for non-response; however, no individuals refused to complete the survey. Limited internet and mobile coverage are likely contributing factors to the low response rate.

In total, 27 of the surveys were administered in Dari, five in Pashto, and two in English.

LIMITATIONS The evaluators faced a variety of common constraints in this evaluation. The evaluation team was only in the field for two weeks, which limited the time available for both key informant interviews and surveys. Accordingly, the evaluation team was unable to reach respondents from the entire SPS chain within the country. Security conditions in Afghanistan created an additional limitation, often constraining mobility of the team while in-country.

Another limitation the team faced was a relatively low response rate on the Trainee Survey. This low response rate poses a threat to the internal validity of the data; therefore, the evaluation team cannot rule out alternative explanations to the findings. Given the low response rate, data from the mini-survey is useful for descriptive purposes only and may not be representative of the perceptions of trainees overall. Through combining various trainee lists and frequent emails and calls, the team obtained responses from 34 out of 107 trainees. In the context of Afghanistan, where official engagement with national government or a foreign entity can be dangerous, this is a strong result.

The evaluators also faced a variety of constraints unique to this FAS program. Most notably, program documentation was sparse and the approved work plan did not contain the program logic, indicators, or targets from which to measure progress. Additionally, the program staff was not available to meet

14 See Annex VI – Data Collection instruments. 15 In total, there were three separate rounds of emails sent to non-respondents.

12


with the evaluation team, no program activities were underway for the team to observe, and no program office exists in Afghanistan at this time.

The evaluation team would also like to acknowledge potential biases in the data. First, recall bias is a common problem in performance evaluations that often require respondents to report on activities and outputs that have occurred over the life of a program in one single instance. Participants may – intentionally or unintentionally – report past events differently. Another limitation to this evaluation is selection bias. The evaluation team selected interviewees non-randomly based on availability and engagement with program activities. Undoubtedly, the biggest source of bias in a performance evaluation is response bias, which stems from respondents misreporting for any number of reasons, generally protecting a relationship with an individual or organization (i.e., USDA, Government of the Islamic Republic of Afghanistan or GIRoA, etc.), telling the evaluator what they want to hear, or competing interests of some kind.

The team sought to mitigate these threats primarily through data triangulation across sources and methods, but also through peer review, debriefing, and member checking. It is worth noting that all of the evaluation findings and recommendations independently articulate, more often than not, several sources contacted during the evaluation. These sources included program trainees, supervisors, counterparts, line organizations, Purdue University staff, and senior ministry officials, including the deputy minister of MAIL.

13


FINDINGS EVALUATION QUESTION ONE Is the SPS Program achieving its specific objectives and to what extent are program indicators being met? The contractor shall consider to what extent program objectives lead towards the achievement of USDA’s Strategic Plan goals and Afghanistan’s assistance framework.

Program Alignment with Strategic Goals The USDA/FAS draft paper entitled “Afghanistan – SPS Capacity Building Development” by USDA/FAS/OCBD stated that “the overall goal of the Program is to strengthen domestic capabilities in safe food production, processing, and inspection; and to ensure safe and wholesome food for the Afghan people” (p. 1). This overall goal stems from a joint, four-day workshop which was held in April 2011 among USDA/FAS, MAIL, and the European Commission. From this workshop, MAIL identified three major SPS priorities for Afghanistan:

1. MAIL wants to improve its surveillance and quarantine procedures so that they are consistent with the international standards. Important international standards include Codex Alimentarius for food safety, International Plant Protection Convention for plant health, and World Organization for Animal Health for animal health.

2. MAIL wants to improve its laboratory diagnostic capabilities in animal health, plant health, and food safety.

3. MAIL wants training in Risk Analysis as it relates to food safety, animal health, and plant health for agricultural products.

In support of the conclusions of this workshop, United States Secretary of Agriculture Tom Vilsack and Afghan Minister of Agriculture, Irrigation and Livestock Mohammad Asif Rahimi wrote an article entitled “Afghan Farmers Will Determine the Success of the Nation’s Economy and Security.” In it, they state that, “Customers know that Afghanistan has the potential to produce high-quality products, but they require the proper support to develop their grading, packaging and sanitary methods per international standards. When this support takes root, Afghan agribusiness will flourish.”

Similarly, MAIL Deputy Minister Ghani Ghuriani confirmed the great need to develop an SPS institution for MAIL and was strongly supportive of the potential for developing an articulated program in SPS that would benefit the country. Other informants within MAIL’s National Agriculture Development Framework echoed the need for a food safety system.

Further, the Director of the Ministry of Commerce’s Export Promotion Agency of Afghanistan (EPAA) (June 19, 2013) reported that the institutional framework for the link between SPS and trade is missing among growers, traders, exporters, and government agencies. The Director continued, “Agricultural products form 70 percent of this country’s exports and there is a strong need to develop the capacity of farmers, supporting agencies, MAIL, and private sector businesses…above all, there is a strong need to coordinate this whole effort among all above players and to provide the necessary infrastructure for pre-harvest, post-harvest, disease diagnostics, and disease control subjects for exportable agricultural

14


products.”16

The evaluation team concluded that the overall objective supports the GIRoA’s priorities and needs.

Relevance of Program Objectives To accomplish these strategic goals, Purdue created three main objectives for the SPS Program:



3. Train MAIL and the DAILs staff in risk management for plant health, food safety, and animal health.

Objective 1 refers to training Afghans in how to develop, in terms of plant pests for example, “strategies and systems that will allow them to intercept exotic insects or pathogens in imports at the border and to detect the presence of invasive species within Afghanistan before the populations become established in large numbers” (p. 5, 1st Purdue work plan).

Objective 2 refers to training people to improve their laboratory skills so as to accurately identify various pests that are present in plants, animals, and food.

Objective 3 deals with risk management, generally defined to include the identification, assessment, and prioritization of risks. Objective 3 depends on the accuracy of the Afghan’s laboratories to identify pests (Objective 2). However, risk management also includes assessing the potential danger to the food supply/exports of these pests and then prioritizing its danger against other SPS threats.

The SPS Program’s objectives are consistent with FAS’ higher goals of establishing domestic, safe food production, identifying opportunities for agricultural exports, and ensuring exports meet SPS standards.

Planned Activities/Implemented Activities During the evaluation time frame, Purdue trained 103 students. It held five workshops, varying in duration from two to five days. In total, Purdue conducted 27 days of training in Afghanistan – 24 in Kabul and three in Herat. Below is a list of activities planned by Purdue in the first work plan and activities implemented by December 31, 2012:

1. Objective 1. Build capacity to implement regulatory systems for quarantine and surveillance, incorporating international standards for plant health, food safety, and animal health into Afghanistan’s laws and regulations:

a. Planned: One-week workshop for middle and upper level officials at MAIL on the rules and functions of the three primary international standards including the International Plant Protection Convention, World Organization for Animal Health, and Codex Alimentarius.

Result: Data gathered through interviews and the document review indicates that this planned activity never occurred.

b. Planned: During and after the visit for the International Standards Workshop, the Purdue SPS

16 Ministry of Agriculture, Irrigation, and Livestock – Islamic Republic of Afghanistan Home Page, from: http://mail.gov.af/en/page/5189, August 6, 2013.

15


team planned to work with senior officials at MAIL and DAIL to identify implementation issues for SPS in Afghanistan.


c. Hazard Analysis Critical Control Points (HACCP) Food Safety

i. Planned: Brief assessment to identify specific food production systems to be targeted for case study development and subsequent HACCP implementation.

Result: reported Purdue University trip13 in April, 2012.

ii. Planned: Workshop for HACCP.

Result: Good Agricultural Practices, Good Manufacturing Practices, and Hazard Analysis Critical Control Points; June 9–13, 2012; Kabul – five days – 33 trainees.

iii. Planned: Using the training-of-trainers model, three additional regions of Afghanistan would be trained in HACCP.


iv. Planned: Two to three Afghan staff will visit Purdue University for two to three weeks for a structured training program.

Result: While it appears the program placed considerable efforts to send Afghan trainees to Purdue, difficulties in obtaining visas frustrated placement. During the evaluation field research phase (June 2012), USDA Kabul officers reported that Purdue was exploring alternative arrangements to place four Afghans at a counterpart university in Turkey for training in Food Safety.

v. Planned: Workshop developing basic food safety capacity in Kabul laboratories.

Result: Basic Sanitation Training; November 17–18, 2012; Kabul – two days – 39 trainees.

2. Objective 2. Improve laboratory diagnostic capabilities for plant health, food safety, and animal health. In these workshops, Purdue planned to modify SPS materials for use in Afghanistan, translate materials into local languages, and teach the skills of training-of-trainers. In addition, two to four Afghan staff for each of the three topic areas (plant health, food safety, and animal health) were to be brought to Purdue University for a structured two to six week training program:

a. Plant Health

i. Planned: Workshop on basic plant health regulatory systems for quarantine and surveillance for middle/upper level MAIL officials.

Result: Data gathered through interviews and the document review indicate that this planned activity never occurred.

ii. Planned: Improve laboratory diagnostic capabilities for plant health.

13 As reported in KII with a MAIL laboratory technician.

16


Result: Plant Diagnostics workshop; April 21–May 2, 2012; Kabul – nine days – 13 trainees; Plant Diagnostics workshop; October 6–10, 2012; Herat – four days – 12 trainees.

iii. Planned: Two to four Afghan staff to visit Purdue University for two to six weeks for a structured training program.

Result: None during evaluation period.14

b. Animal Health

i. Planned: Training in diseases in horses, cattle, sheep, goats, and key wildlife diseases relevant to Afghanistan.

Result: Poultry workshop; Real-Time: AIV and NDV; June 30–July 4, 2012; Kabul – five days – nine trainees.15

Result: Data gathered through interviews and the document review indicates that workshops for diseases in horses, cattle, sheep, goats, and key wildlife diseases relevant to Afghanistan did not occur.

ii. Planned: Two to three Afghan staff to visit Purdue University for two to three weeks for a structured training program.

Result: Data gathered through interviews and the document review indicate that this planned activity never occurred, but the evaluators note that the program devoted considerable energy to overcome visa restrictions.

3. Objective 3. Train MAIL and the DAILs staff in risk management for plant health, food safety, and animal health.

a. Planned: Train middle/upper level MAIL and the DAILs staff in risk management for plant health, food safety, and animal health. Develop and present training materials to top level directors responsible for policy implementation.


b. Planned: Stakeholder involvement in problem identification, planning, implementation, and evaluation.

Result: Purdue is working with senior level people such as Deputy Minister Ghuriani in MAIL and many of the Heads of Directorates in that Ministry.

Result: There is some evidence that Purdue is working sporadically with other stakeholders in the SPS system, such as exporters and importers of agricultural products, the agricultural faculties at Afghan universities, Afghan food processors located, the Ministry of Finance, NGOs, and others involved in SPS.

During the evaluation period, Purdue University devoted nearly all its efforts to Objective 2: Improve laboratory diagnostic capabilities for plant health, food safety, and animal health. The program conducted 14 While outside the evaluation time frame, the team notes that one MAIL staff left mid-June 2013 for a one month period at Purdue University’s Plant Diagnostic Laboratory in the US. 15 While outside the evaluation time frame, Purdue conducted a Poultry Workshop: Real-Time for Detection of NDV, ILTV, and IBDV; February 23-27, 2013; Kabul – four days – 3 trainees.

17


five workshops – two in plant health, two in food safety, and one in animal health in Afghanistan.

The program made no progress in the implementation of the activities associated with building regulatory systems, including policy interventions, under Objective 1, or training middle and senior managers in risk management techniques as per Objective 3.

Table 1 shows planned activities outlined in the first SPS Program work plan dated September 9, 2011.

18


Table 1 – Planned Program Activities

Objective Activity Duration Trainees Number Location Planned DateRisk Management International Standards Workshop - Int'll Plant Protection Con., World Organization for Animal Health & Codex Alimentarious one week middle/snr MAIL mgt not stated Kabul 2012 (first year)

All Work with MAIL staff to modify and translate materials and guides into local lanaguage not stated na Kabul, presumablyAll Core group of MAIL staff who undertake training will receive ToT not stated not stated Kabul, presumablyAll Structured 2-6 week training at Purdue University in US: Plant Health, Animal Health & Food Safety 2-6 weeks 2-4 students: each branch 6 to 12 US not stated

Plant Health Workshop on Basic Plant Regulatory Systems for Quarantine and Surveillance not stated middle/snr MAIL mgt not stated Kabul 2013 (second year)Plant Health Improved Laboratory Diagnostic Capabilities insect id & plant disease diagonosis not stated core group - MAIL staff not stated Kabul 2012 (first year)Plant Health Improved Laboratory Diagnostic Capabilities - important groups of pests and diseases found in Afg & border not stated core group - MAIL staff not stated Kabul 2013 (second year)Plant Health Basic diagnostic training to MAIL & DAIL staff in provinces not stated MAIL not stated to be confirmed 2013 (second year)Plant Health Expand diagnostic training (as above) to other provinces not stated MAIL not stated to be confirmed 2014 (third year)Plant Health Visiting Scholars - key MAIL staff brought to US for training and experience not stated MAIL not stated US - Purdue not statedFood Safety Workshop on HACCP - needs assessment not stated not stated Kabul not statedFood Safety Workshop on HACCP - training including ToT not stated MAIL / DAIL not stated four regions Afg not statedFood Safety HACCP - training in US not stated MAIL / DAIL 2 to 3 US - Purdue not statedFood Safety HACCP - above trainees return & train HACCP not stated MAIL / DAIL 3 to 3 Afghanistan not statedFood Safety HACCP - returned US trainees implement HACCP for target food & ag products not stated MAIL / DAIL not stated Afghanistan not statedFood Safety Develop lab capacity in Kabul - create prototype for basic food safety testing lab not stated not stated not stated Kabul, presumably not stated

Animal Health Workshop - animal health topics - first serial one week MAIL not stated Kabul, presumably not statedAnimal Health Workshop - animal health topics - second serial one week MAIL not stated Kabul, presumably not statedAnimal Health Workshop - animal health topics - possible third serial one week MAIL not stated Kabul, presumably not statedAnimal Health Training - management of reference animal disease diagnostic laboratory 2-3 weeks MAIL / DAIL 2 OR 3 Kabul, presumably not stated

Risk Management Risk Analysis Workshop one week middle/snr MAIL mgt not stated Kabul, presumably not stated

19


EVALUATION QUESTION TWO Is the SPS Program reaching targeted beneficiaries? If not, why not? If yes, what are the results?

Participants, MAIL managers, and program instructors find the program selection criteria for beneficiaries/trainees to be clear and verifiable. The data indicate that targeting under Objective 2 was successful, but targeting under Objectives 1 and 3 cannot be assessed at this time, as no activities under these areas occurred during the evaluation time frame. The evaluation team could not determine whether the program achieved the targeted number of trainees, as the work plan did not set such targets.16

A review of engaged beneficiaries, as per SPS Program objectives, once again indicates that virtually all activities focused on the development of MAIL laboratory diagnostic capacity (Objective 2).

Objective 1: Build capacity to implement regulatory systems The evaluation team, through key informant interviews, the trainee survey, program reports, or other material, could not find any evidence of the program developing capacity within MAIL staff or other beneficiaries to implement regulatory systems.

Objective 2: Improve laboratory diagnostic capacity The evaluation team found that virtually all implemented activities, therefore all targeted beneficiaries, reside within this objective.

Within the evaluation time frame, from October 2011 to December 31, 2012, the program conducted five workshops.

Plant Health 1. Plant Diagnostics April 21–May 2, 2012 Kabul, Afghanistan 13 trainees 2. Plant Diagnostics October 6–10, 2012 Herat, Afghanistan 12 trainees

Animal Health 1. Real-Time: AIV and NDV June 30–July 4, 2012 Kabul, Afghanistan 9 trainees

Food Safety 1. HAACP/GMPS/GAPS June 9–13, 2012 Kabul, Afghanistan 33 trainees 2. Basic Sanitation Training Nov. 17–18, 2012 Kabul, Afghanistan 39 trainees

Objective 3: Train MAIL and DAIL staff in risk management The evaluation team, through key informant interviews, the trainee survey, program reports, or other material, did not find any evidence of the program training MAIL/DAIL staff in risk management procedures.

Training Survey Results In order to assess the extent to which the program successfully targeted trainees, the evaluation team created a post-training survey. While the evaluation team found that most trainees appreciated the provided laboratory training, at least five respondents noted that there were insufficient supplies and reagents in workshops to facilitate learning. One scientist at CVDRL, the national veterinary diagnostic

16 The second approved work plan, dated December 6, 2012, and enacted in January (outside the evaluation time frame), is specific in setting goals for the number of beneficiaries.

20


laboratory,17 said it more succinctly: without the proper equipment and reagents (and a sufficient supply of reagents to use several months after the training), all trainings were “a waste of time” (June 17, 2013).

Table 2 displays results from the Trainee Survey. A total of 33 trainees completed the survey –seven in animal health, 15 in plant diagnostics, and 11 in food safety. Results for each of the 12 survey questions are below, disaggregated by workshop type. The right-most column presents the total for all three workshop types.

The data indicate that trainees were generally satisfied with the workshops. Overall, 73 percent of trainees indicated that they received a briefing on the purpose of the course, 79 percent indicated that material organization was “very good” or “excellent,” 94 percent expressed that the pace of instruction was “just right,” and 76 percent rated the quality of the training as “very good” or “excellent.”

The survey also asked participants a series of questions related to the language of instruction of the workshops. While 30 of 33 respondents (91 percent) indicated that the predominant language in the course was English, 85 percent stated that the language of instruction did not negatively affect learning. Moreover, 70 percent indicated that the language of printed materials did not affect the difficulty of reading. While these results suggest that language was not a significant barrier in the workshops, the remaining 30 percent of trainees affirmed that printed materials were difficult to read due to language.

The survey also contained several questions related to use of skills and knowledge acquired in trainings (questions 6 and 10–12). A total of 61 percent of trainees indicated that they practice these skills daily and 33 percent indicated that they practice these skills weekly, which suggests that the training is highly relevant to their daily work. Responses to the question, “How able are you to integrate the training in your work?” reflected this finding, with only three percent indicating “not at all.” Of note, food safety trainees reported being able to apply the training to their work overwhelmingly less than animal health or plant diagnostics trainees. While we must caution against drawing conclusions from data with such a small sample size (11 for food safety), this finding may indicate a lack of relevance of this curriculum to those targeted and recruited participants. Nevertheless, 85 percent of trainees reported that their departments increased SPS activities since they participated in training.

Table 2. Trainee Survey—Satisfaction with Workshops, by Topic

17 Central Veterinary Diagnostic and Research Laboratory in Kabul.

Respondents Workshop % Respondents Workshop % Respondents Workshop % Respondents Workshop %Yes 6 86% 9 60% 9 82% 24 73% YesNo 1 14% 6 40% 2 18% 9 27% No

Animal Health Plant Diagnostics Food Safety All Work Shops Combined 1 - Were you briefed on the purpose of this course?

Respondents Workshop % Respondents Workshop % Respondents Workshop % Respondents Workshop %Excellent 0 0% 7 47% 2 18% 9 27% Excellent

Very Good 6 86% 6 40% 5 45% 17 52% Very GoodGood 0 0% 2 13% 3 27% 5 15% Good

Fair 1 14% 0 0% 0 0% 1 3% FairPoor 0 0% 0 0% 0 0% 0 0% Poor

Animal Health Plant Diagnostics Food Safety All Work Shops Combined 2 - How well was the course material organized?

21


Respondents Workshop % Respondents Workshop % Respondents Workshop % Respondents Workshop %Just Right 7 100% 15 100% 9 82% 31 94% Just Right

Too Fast 0 0% 0 0% 2 18% 2 6% Too FastToo Slow 0 0% 0 0% 0 0% 0 0% Too Slow

Animal Health Plant Diagnostics Food Safety All Work Shops Combined 3 - Describe the pace of instruction.

Respondents Workshop % Respondents Workshop % Respondents Workshop % Respondents Workshop %Sufficient Access 6 86% 12 80% 1 9% 19 58% Sufficient Access

Some Access 1 14% 3 20% 1 9% 5 15% Some AccessNo Access 0 0% 0 0% 9 82% 9 27% No Access

Animal Health Plant Diagnostics Food Safety All Work Shops Combined 4 - Did you have sufficient access to lab facilities for the course?

Respondents Workshop % Respondents Workshop % Respondents Workshop % Respondents Workshop %Sufficient Access 6 86% 12 80% 2 18% 20 61% Sufficient Access

Some Access 1 14% 3 20% 6 55% 10 30% Some AccessNo Access 0 0% 0 0% 3 27% 3 9% No Access

Animal Health Plant Diagnostics Food Safety All Work Shops Combined5 - Did you have sufficient access to supplies and equipment for the course?

Respondents Workshop % Respondents Workshop % Respondents Workshop % Respondents Workshop %Daily 0 0% 12 80% 8 73% 20 61% Daily

Weekly 6 86% 3 20% 2 18% 11 33% WeeklyMonthly 1 14% 0 0% 0 0% 1 3% Monthly

Never 0 0% 0 0% 1 9% 1 3% Never

Animal Health Plant Diagnostics Food Safety All Work Shops Combined6 - How often do you practice the skills learned in the course?

Respondents Workshop % Respondents Workshop % Respondents Workshop % Respondents Workshop %English 7 100% 12 80% 11 100% 30 91% English

Dari 0 0% 3 20% 0 0% 3 9% DariPastu 0 0% 0 0% 0 0% 0 0% Pastu

Animal Health Plant Diagnostics Food Safety All Work Shops Combined7 - What was the predominent language in the course?


Animal Health Plant Diagnostics Food Safety All Work Shops Combined8 - Did the language of instruction negatively affect your learning?


Animal Health Plant Diagnostics Food Safety All Work Shops Combined9 - Did you have difficulty reading the printed material because of language?

Respondents Workshop % Respondents Workshop % Respondents Workshop % Respondents Workshop %Completely 5 71% 9 60% 2 18% 16 48% CompletelySomewhat 2 29% 6 40% 8 73% 16 48% Somewhat

Not at All 0 0% 0 0% 1 9% 1 3% Not at All

Animal Health Plant Diagnostics Food Safety All Work Shops Combined10 - How able you to integrte the training in your work?

22


Integration of training into the work place is the key objective of training interventions. Chart 1 shows reasons why participants could not integrate what they learned in workshops into their jobs (respondents were allowed to only pick one of the four choices). Thirty-three percent lacked the equipment and supplies in their labs, three percent said it was not part of their jobs, and the remaining percentage of respondents had other reasons.

Chart 1. Trainee Survey – Reasons Participants cannot Integrate Material

The team also asked respondents an open-ended question about how courses could improve. Chart 2 lists the most prominent suggestions. Nearly all respondents had multiple suggestions, all of which were coded as part of the analysis. The most frequent suggestion was to extend the length of the courses (39 percent). Some respondents explained that they just started to understand the material and see its applicability when the course ended. A Purdue faculty member confirmed this opinion, stating that short-term trainings were only marginally effective, e.g., “we train for a week or more, then we leave.”

The next two most prominent responses were requests for more equipment – both during the workshop and in their workplace labs – to better learn and utilize the information (27 percent) and that the courses should be more practical and less theoretical (27 percent).

Additionally, 21 percent of the respondents indicated that they want more courses. Though the SPS courses are valuable, offerings are too infrequent. Eighteen percent of the suggestions encouraged the presentation of the lectures and material in local languages, such as Dari. Also, 18 percent of the people


Animal Health Plant Diagnostics Food Safety All Work Shops Combined11 - Since your training, has your department increased its activities in SPS?

Respondents Workshop % Respondents Workshop % Respondents Workshop % Respondents Workshop %Excellent 0 0% 3 20% 3 27% 6 18% Excellent

Very Good 6 86% 9 60% 4 36% 19 58% Very GoodGood 0 0% 2 13% 4 36% 6 18% Good

Fair 1 14% 1 7% 0 0% 2 6% FairPoor 0 0% 0 0% 0 0% 0 0% Poor

Animal Health Plant Diagnostics Food Safety All Work Shops Combined12 - How would you rate the quality of the training?

0%

5%

10%

15%

20%

25%

30%

35%

Not part of job No support fromboss

Lack of equipmentor supplies

Other

23


thought that there should be more field trips to visit their regional labs and other labs to see the applicability of the information at those locations.

There were some specific suggestions that may be helpful to Purdue in the future, though not frequently noted. Two people expressed an interest in expanding the avian focus of the animal health workshops to include large animals and small ruminant animals. One person suggested that it would be helpful if Purdue could provide enough supplies to last three to four months so that students can practice what they have learned. Some respondents from the plant diagnostic workshop indicated a desire to learn about the safe and appropriate use of pesticides.

Chart 2. Trainee Survey - Suggestions to Improve Courses

EVALUATION QUESTION THREE What internal and/or external factors are positively or negatively affecting implementation? How can these factors be further developed or overcome, respectively?

The evaluation team defines external factors as those factors outside of Purdue University’s control and internal factors being those within Purdue’s control.

Positive External Factors (outside Purdue’s control) • The professional staff members at MAIL are motivated to participate in SPS programming. They

understand the importance of SPS in the development of a safe food supply in Afghanistan and increasing its country’s exports. Deputy Minister Ghuriani was especially effusive about the value of SPS to MAIL.

• Similarly, the participants at the five workshops were motivated to learn about SPS.

Negative External Factors (outside Purdue’s control) • The operating environment in Afghanistan is exceptionally difficult. Routine security incidents

threaten the physical safety of program activities and participants and pose severe logistical challenges. As a result, Purdue encounters difficulty in recruiting staff to teach workshops in

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

Longercourses

Morepractical

Moreequipment

Morecourses

Field trips(labs)

Locallanguages

24


Afghanistan. In addition, there are significant difficulties for Afghans to obtain visas to visit the U.S. for training.

• Lack of equipment and supplies for laboratories are almost a universal problem at MAIL and MoPH. Furthermore, the staff members who operate these labs have varying levels of professional training and background. Similarly, there is a marked difference between MAIL and MoPH’s baseline capacity in these areas. Purdue instructors face a daunting task of deciding at which level to teach their classes (e.g., to the best trained students or least trained).

• The lack of interagency cooperation hinders coordination of SPS activities, especially when topics such as food safety are important to different ministries.

Positive Internal Factors (within Purdue’s control) • The Afghan students indicate a high-level of satisfaction with the workshop teachers. The teachers

delivered high-quality instruction that motivated the students to learn.

• Purdue’s extensive experience in Afghanistan and in MAIL allows its team to quickly contact key people within the Afghanistan agricultural industry. Purdue also understands the relationship of MAIL and DAIL (provincial and local agricultural offices), so they can develop programs for both audiences with relative ease.

Negative Internal Factors (within Purdue’s control) • The language and material of instruction is predominately in English. While most survey feedback

from the workshop participants was positive (note Table 2), there were problems associated with the reliance on English. As noted previously, English-language materials are often problematic as some students are unable to review the material after the class has ended, thus limiting their ability to incorporate this new knowledge and skills in their jobs.

• Purdue did not hire a Kabul-based staff member to coordinate activities for the SPS Program. As a result, workshops (while well received) are too infrequent, perhaps because of coordination issues in Kabul. SPS requires interagency coordination and intra-agency cooperation. In order to develop a sustainable SPS infrastructure, there must be a network of concerned professionals willing to move the SPS agenda forward, creating an informal institution in the absence of a formal one.

• The workshops are, on average, too short. Perhaps this is due to US-based faculty responsibilities at their home institutions. Numerous key informants indicated that they wanted trainings to last several weeks. Students said that they wanted longer workshops so there would be greater opportunities for review at the end of the course. In general, the courses lacked sufficient time for review and practice.

EVALUATION QUESTION FOUR How is the sustainability of the SPS Program being addressed? Are the beneficiaries being followed up with in order to track their progress?

The issue of sustainability is particularly salient given the imminent disengagement of USDA FAS and other USG-funded programming in September 2014. The evaluation team defines sustainability in this program as the establishment of permanent capacity within MAIL to implement and manage SPS and food safety training, systems, and regulation. The program’s three objectives are interlinked and critical to support the establishment of a functioning and sustainable SPS system. As the program only trained laboratory technicians under Objective 2 so far, achievement of this goal remains incomplete.

The evaluation team did not find much evidence that the program engaged with senior MAIL

25


management – at the Director General level and above – to support the establishment of an institutional framework to manage and promote an SPS mechanism within the Ministry. Recognizing that the program incompletely implements its mandate, the evaluation team found that the laboratory and food safety training, educating over 100 specialists in various disciplines through five workshops, had a positive cumulative effect. Trainee graduates increased their skills as a result of the program’s courses. For instance, some of the participants took both workshops in their field of study (e.g., plant health, animal health, or food safety) so as to develop their personal expertise and then to contribute to their department’s productivity.

Purdue designed the workshops with utility in mind. Participants said workshops were especially valuable if two were taken together, as the course content proved to be mutually supportive and complementary. Some of the trainings introduced best management practices and standard operating procedures in order to encourage sustainable practice of such procedures by those newly trained individuals upon return to their own laboratories.

Individual instructors established professional relationships with trainees. These instructors became ad hoc points of contact to the Kabul graduates. For example, the head of the Kabul Plant Diagnostic Laboratory (June 17, 2013), MAIL scientist Abdul Babori, sends digital pictures of diseases and other pests to the Purdue Plant Diagnostic Laboratory for help in diagnosis and to learn about recommended control practices. Similarly, people in the Animal Health Lab maintain communication with Dr. Ramesh Vermulapalli (June 15, 2013).

The evaluation team received conflicting messages about the ability of the participants to apply the training content. The data in Table 1 indicate that the training content is useable in the participants’ jobs; however, a third of survey respondents and most people directly interviewed indicated that due to the lack of equipment and supplies in their laboratories, they cannot apply lessons learned in the trainings. This may suggest that despite the positive reception, lack access to adequate equipment and supplies limit the application of the training content.

At the program level, it does not appear that the Purdue SPS Program operates sustainably. As mentioned before, the program only progressed toward one of its three Objectives. So far, the activities target staff/technicians/scientists and not middle level or upper level management.

The work plan stated that middle level or upper level management will receive training to learn strategies to regulate SPS and to understand risk management so as to develop systems that identify, assess, and prioritize threats to SPS in Afghanistan under Objectives 1 and 3. Both Objectives 1 and 3 support the institutionalization of SPS, particularly within MAIL. This lack of training for middle and upper level management is confounded by the hierarchical nature of ministries in Afghanistan. A lack of approval by supervisors may stymie lower-level staff’s advancement in the SPS realm.

Follow-Up with Beneficiaries The team found that the program does not systematically follow-up with students or track their progress beyond their participation in the workshops. Other than the ad hoc, professional relationships that exist between instructors and their students, no procedures for follow-up are in place. One MAIL department chair noted (June 19, 2013) that the program does not maintain contact trainees within MAIL either.

Establishment of Regulatory Systems in MAIL As discussed earlier, a key component to promote SPS sustainability is the establishment of regulatory systems within MAIL and the required management mechanisms within the Ministry. Discussions with senior level MAIL staff indicate a strong desire to have an institutional expert to analyze the present system, advise on improvements, and coach senior staff to make the changes required in institutional

26


structures (KII, June 24, 2013). Given the imminent cessation of USG-funded assistance in the fall of 2014, it is important to build the enthusiasm for and capacity to develop a functioning regulatory system while FAS still has an in-country presence.

EVALUATION QUESTION FIVE The SPS Program was established to train staff via training-of-trainers and direct methods. What evidence is there that this is having a sustained effect? How can it be improved?

In the five workshops taught so far, there was no a formal component for training-of-trainers. However, some of the graduates subsequently taught material in other locales. For example, Abdul Babori and Zakrhia Faizi in the Plant Protection Department helped teach the Plant Diagnostic workshops in Kabul and Herat (approximately two days at each workshop). The Purdue instructors were supportive of their efforts. Also, after the two workshops, Babori trained 120 farmers in Laghman Province, 120 farmers in Nuristan, 120 farmers in Nangarhar, and 120 farmers in Kunar in disease diagnosis. In addition, he trained 24 DAIL extension agents in the same subject in the four above-mentioned provinces. Finally, he conducted a one-day training for 80 students and faculty from Kabul University. In this case, his mentoring by Purdue instructors was helpful in sharing the information.

Nevertheless, the evaluation team heard almost universally that there was not a training-of-trainers component in the courses among the rest of the students. As a laboratory manager who participated from the Animal Health training said, “I can share the information I learned in the course with my colleagues, but I really do not know how to teach it effectively.” He went on to say: “I would like to be taught how to do training-of-trainers.” The lack of training-of-trainers is most likely associated with the brevity of the courses where the Purdue instructors only have time to teach the essential material for their workshops.

EVALUATION QUESTION SIX How has the SPS Program contributed to the creation and implementation of a functioning SPS system and what needs to happen to improve/expand that benefit?

The three program objectives18 are inter-related and key to a functioning SPS system in Afghanistan. As Purdue only worked to improve laboratory diagnostic capacity at MAIL so far – Objective 2 – effective contributions to a functioning SPS system are not apparent. Most activities targeted the practitioner or laboratory specialist level. The evaluation team did not find evidence that the SPS Program engaged with senior MAIL management – at the Director General level and above – to support the establishment of an institutional framework to manage and promote a SPS mechanism. Nevertheless, at the laboratory level, the training of technicians will doubtlessly have an impact. Five workshops, with complementary themes, had a cumulative effect in MAIL and MoPH, with over 100 people receiving intense instruction on SPS. As a result people are aware of the importance of SPS and have increased diagnostic and laboratory skills.

EVALUATION QUESTION SEVEN

18 1) Build capacity to implement regulatory systems for quarantine and surveillance incorporating international standards for plant health, food safety, and animal health into Afghanistan’s laws and regulations; 2) Improve laboratory diagnostic capabilities for plant health, food safety, and animal health; and 3) Train MAIL and the DAILs staff in risk management for plant health, food safety, and animal health.

27


How are the implementers working to identify participants? Are they in constant contact with the Ministry?

Purdue worked closely with MAIL counterparts to identify and select trainees based on an agreed upon criteria. Once the criteria are agreed upon, then MAIL makes the actual selection. Also, as noted by a scientist in animal health (June 15, 2013), Dr. Ramesh Vermulapalli, he and other Purdue scientists met with local counterparts at the Central Veterinary Diagnostics and Research Laboratory to determine the existing capacity of the facility so that Vermulapalli could design his course appropriately.

The evaluation sought to learn if collaboration and consultation with MAIL counterparts informed training plans. The evaluation team found that this was generally not the case. Comments from one senior ministry employee suggested a less collaborative approach to curriculum selection, (June 24, 2013):

“The future training agenda should be according to our needs. In the previous training, they [Purdue] told us they want to do certain training. We tell them we'd prefer they teach more and something else. They [Purdue] say, this is the training we are going to conduct, and then we [MAIL] comes to the conclusion that something is better than nothing and we say okay.”

One of the key informants from MAIL (June 20, 2013) indicated that he had been to two workshops, which were good, but that neither himself nor anybody else he knew had a clear understanding of what was going to happen next. As noted previously, external factors limit the program’s ability to bring professors to Afghanistan or to bring Afghans to the Purdue campus for training in the U.S. This limited collaboration is also likely a result of the absence of SPS activities at the middle and senior management level, since these players would presumably play a role in guiding the strategic direction of SPS within MAIL. These data from key informant interviews suggest a more dominant role for Purdue in critical decision making with minimal input from MAIL.

OTHER FINDINGS

Work Plan The evaluation team noted that the first work plan (approved September 9, 2011) was vague in setting specific targets of trainee numbers. This is understandable as most first work plans in Afghanistan require appreciable revision as the program copes with complex, changing field conditions. A revised, approved work plan19 was indeed more specific in its listing of targets, indicators, and metrics, though it should be noted it was approved some 15 months after the program was launched. As such, the program operated under an ill-defined work plan for almost half the program’s duration.

Field Program Management The program did not have a permanent presence in Kabul, leaving Purdue to coordinate, manage, and arrange the vast majority of activities from the U.S. This off-site coordination – rare for a capacity building program of this scale in Afghanistan – presented program challenges, particularly in coordinating workshop logistics.

19 Second work plan was approved December 6, 2012, and was effectively enacted in January 2013, thus outside the evaluation time frame.

28


For instance, in at least one workshop, an instructor from Purdue brought reagents – critical testing materials – in his luggage. This frustrated a Kabul laboratory manager who said there were not enough reagents to use for post-training practice. He said Purdue could have, instead, used the established government system of ordering sufficient materials through a reference laboratory in the UK for routine, direct delivery to the Kabul lab before the course began.

Key informants in Kabul, from senior MAIL managers to USDA counterparts, as well as USDA colleagues in Washington, D.C., said that the lack of Purdue field staff hampered program coordination and follow up. One USDA officer went on to say that not having a full-time Kabul program staff member has been “catastrophic.”

Implementation Rate From September 2011 to December 31, 2012, the program trained 107 people through five workshops for a total of 27 training days in Afghanistan. As the first work plan was not sufficiently detailed to precisely determine whether the program is implementing to schedule, the evaluation team analyzed the expenditure rate. Expenditures, the team feels, can be one proxy indicator of activity, particularly if the line items correspond with deliverables.

The program budget estimated expenditures on an annual basis. The evaluation team, breaking these expenditures into monthly units, estimated the program budgeted to spend $854,000 by December 31, 2012.20 According to the September–December quarterly report, the SPS Program expended $446,042 of its budget by the end of 2012. In other words, the SPS Program may have only spent half of its planned budget by year-end 2012.

Request for Training & Equipment The evaluation team was struck by the passion that program trainees and MAIL management harbor for SPS. In almost every discussion, interviewees suggested or proposed further training and requested modest amounts of equipment or supplies. According to MAIL scientists (June 17, 2013), most provinces do not have sufficient laboratory equipment and supplies to utilize SPS training in their work. This is the crux of the challenge that Purdue faces – the vast need for training (both intensive and extensive) contrasted by the lack of facilities for Afghans to learn and use these new skills.

Representative requests:

• A senior official at the Export Promotion Agency (June 19, 2013) suggested that they have a training facility that Purdue could use to give exporters practical training in visual inspections that denote obvious problems. Another scientist at the same institution recommended the establishment of a model border inspection post that could also serve as the training vehicle for SPS (June 17, 2013).

• A MAIL scientist at the Central Veterinary Diagnostics and Research Laboratory (CVDRL) suggested that Purdue should conduct workshops and trainings in the seven border zones (June 17, 2013).

• MAIL has requested that the program train the Ministry’s 13 food inspectors stationed in Kabul (June 22, 2013).

20 Breaking an annual budget into average monthly expenditures does not generally reflect a field implementation cycle of minimal/maximal activity; however, taken over the 15-month duration of the evaluation time frame, it averages out somewhat and is a useful, though not exact, indicator.

29


• A MAIL middle manager requested training for the customs inspectors at Kabul Airport and the four major land entry points for imports and exports (June 19, 2013).

• MAIL Acting Deputy Minister Amanuddin Haidary said that the GIRoA system for border inspection for food goods is completely under-developed. He said there are eight border labs which are under-equipped, rendering it extremely difficult for inspectors to take, much less analyze, samples. He requested that the SPS Program provide the training and equipment for two border control labs. In addition, he has pledged, on behalf of the GIRoA, upkeep and staff salaries. He estimated equipment costs would be less than $50,000 per unit (June 23, 2013).

30


CONCLUSIONS All conclusions are based upon evaluating activities from the start of the SPS Program to December 31, 2012, using the first work plan, approved September 9, 2011,21 as a key reference point. Other key sources are three final quarterly reports, detailing activities until end of year 2012. These documents were all complemented with interview data from direct field research.

1. Program training of excellent quality, but challenges remain The evaluation team concludes that the training delivered was of excellent quality and directly on target to program and student need.

The duration of course instruction, usually between two and five days, has been too short leaving insufficient time for student confirmation and review. These trainings should be coupled with longer-term training so that the trainees gain a fuller understanding of the whole SPS process.

One workshop in food safety, well attended and generally well received, was merely two days in duration. Considering the expense and other resources necessary to transport a US-based instructor to Afghanistan, it would have been optimal either to extend this course, to explore topics in greater depth, or to host numerous serials to enable many more food professionals to attend. Furthermore, the dearth of training supplies and materials, coupled with an absence of course material translated into a local language, render the student learning experience needlessly challenging.

2. Program has made progress, but has not met work plan deliverables Afghanistan is an extremely challenging place to implement SPS programming. It is also difficult to recruit specialist staff to teach in Afghanistan – indeed one key informant said they were happy to teach one course in Kabul, but their spouse would not let them return for a second round.

During the evaluation time frame, accounting for 15 months of a 36-month SPS Program, Purdue delivered five courses training 107 people for a total of 27 training days in Afghanistan. Based on its work plan, the program has not produced deliverables for the number of trainings.

Except for one workshop held in Herat, the SPS Program has not expanded beyond Kabul. This is understandable as it may reflect Purdue’s intent to establish a critical base of skills in the capital before branching out. It may also reasonably reflect the reality that security in Afghanistan has appreciably deteriorated since the start of the SPS Program. The SPS Program was not planned, resourced, or structured to operate within a threat environment, which now covers much of the country.

3. Program Management The present off-site management model, where the program is effectively run from the U.S. offices of Purdue University, is inefficient and jeopardizes program effectiveness.

4. Implementation Rate Should Purdue maintain its current implementation rate of activities, it will likely only expend half of its budget by program end in October 2014. To date, Purdue primarily relied on U.S. university instructors to deliver programming; this may contribute to the slow pace of program delivery. 21 The current, second work plan, approved December 6, 2012, did not have any obvious effect or impact on program activities during the remaining three weeks of 2012, so it is not directly considered.

31


5. Sustainability The program did not effectively engage middle and senior level managers at MAIL to help educate, mentor, and support the establishment of an operational SPS mechanism in the ministry. Sustainability is highly unlikely unless management engages in the process.

6. Program of use, but will not reach strategic intent unless activities address all objectives

To date, Purdue almost exclusively focuses on developing laboratory diagnostic skills (Objective 2) and has made almost no progress in the other objectives of building regulatory capacity and risk management systems (Objectives 1 and 3, respectively).

While the building of laboratory diagnostic capacity provides significant, welcomed benefit to MAIL, ultimately the program will not achieve its strategic intent unless it also concentrates on developing capacity regulatory systems and risk management.

While beyond the mandate of this evaluation, it should be noted that in Purdue’s second approved work plan of December 6, 2012, the program has indeed planned activities addressing those two objectives (regulatory systems and risk management) though none have been implemented as of July 2013.

32


RECOMMENDATIONS Once again, it should be noted that the program has made a substantive contribution in developing the laboratory diagnostic capacity within MAIL. Beyond that (second) objective, the evaluation team is confident that there is still time and resources left to bring the program back toward achieving its full potential.

The evaluation team primarily limits its comments to activities drafted in the first work plan, though it appreciates that the program’s modality and effectiveness have not substantively changed to present date. So while appreciating that there have been some positive, planned changes in the work plan approved December 6, 2012, the evaluation team is confident the following recommendations are still valid as of writing (July 2013).

1. Program Review & Revision USDA program management, in consultation with Purdue and the MAIL, should review the original agreement to:

a. Confirm deliverables it has sufficiently covered;

b. Determine areas that need to be addressed (i.e., Objective 1 and 3);

c. Assess available resources and time and also provide a cost estimate for every activity;

d. Conduct a Kabul-based survey to identify activities that are within resource and program objectives; and

e. Confirm work plan activities with MAIL and USDA.

Within this review, the evaluation team recommends addressing or incorporating:

• Establishment of inter-departmental SPS mechanism: As requested by MAIL and in line with Objective 1, the program should support the Ministry in the establishment of an inter-departmental SPS mechanism to ensure quality control, access to resource, and a standardized and rationale training regime. Within the time frame of the SPS Program, an SPS institutional specialist could be placed in Kabul for four months to conduct a detailed analysis of the Afghan food and supply MAIL and other actors with practical recommendations and a plan to move forward.

• Risk Management practice: The SPS Program, as per Objective 3, should conduct specific training as well as less formal mentoring in risk management practice as it relates to SPS in Afghanistan. This will require training of middle level management to better utilize SPS resources and mentoring of senior level managers to ensure resource allocation and inter-agency coordination.

2. Revised Training Approach The evaluation team recommends that the SPS Program:

• Ensure training is appropriate to needs and resources: Some of the training used materials, reagents, equipment, or specialist expertise that are not readily available at MAIL or MoPH. Consequently, the team recommends that training design be calibrated to:

˗ Existing resources and need;

˗ Fall in line with the strategic intent of the SPS Program and host ministry; and

˗ That a reasonable number of graduates can be expected to use the training in their work.

33


• Provide sufficient supplies and basic equipment to facilitate training: The SPS Program has an appropriate budget of $10,000 per training activity. The evaluation team recommends that reagents and other supplies be in sufficient quantity to enable at least one month’s post-course practice. The SPS Program should work closely with MAIL and other counterparts to order these materials in advance. In some, usually provincial sites, the SPS Program should provide as required critical, user-friendly equipment such as microscopes and complementary training.

• Longer courses/more time for review: Most trainee participants report that there was insufficient time for review and confirmation. Wherever appropriate to subject matter, workshops should be at least five days or one work week in length, allowing much more time for practical review.

• Make instructor time on ground more productive: Great value is lost when an instructor is flown into Kabul to teach for just two days. It either limits the material that can be taught or the number of students reached. Additionally, it does not take into account the effect that arduous travel and jet lag has on the instructional ability of the specialist. Thus, in rare cases where two-day workshops are practical, at least three other workshops or courses should be held in one trip.

• Instructor trips should be two full weeks in duration: By ensuring that instructor time on the ground in Kabul is 12 days, the trainer or mentor will be better prepared and rested, leaving more time to teach courses that are more appropriately structured to increase learning.

• More frequent training: During the evaluation period, the SPS Program has held an average of one training session every three months or so. The evaluation team believes that regional specialists could be brought in every four to six weeks to deliver a two-week substantive training course. Meanwhile, twice a month, a local Program Officer (see below) would be able to deliver two or three-day trainings on more general topics. Also, a Program Officer could host introductory instruction and/or qualification testing to select candidates for more advanced training.

• Translate essential course material into local language: As the English language capacity of participants is variable, essential course content should be translated into Dari and Pashto. While this requires some advance preparation, it is not expensive or complicated to have material translated.

• Training-of-trainers: Post-program sustainability is at risk unless participants are able to communicate their knowledge of SPS. The team recommends that a quarterly training-of-trainers program be held to teach professionals how to instruct and disseminate the knowledge.

• Keep training and other activities in Afghanistan: MAIL management has said that it is against sending its staff to other countries for advanced training because of limited professional benefit and the absence of key colleagues causes operational issues at the agency. Experience has certainly shown that it is progressively difficult to obtain visas or organize foreign training for Afghan participants. MAIL requests – and the evaluation team recommends – that the vast majority of training be held in Afghanistan. A focus on local training would enable a greater, more diverse number of Afghans to participate, thereby encouraging sustainability. More frequent training will inevitably reach out and train more Afghan professionals and open the training up to related professionals from Customs, the private sector, etc.

3. Open Recruitment to all Professionals To date, most SPS Program instructors have been recruited from current Purdue professors or from the greater university community. As the SPS Program is not achieving its objectives and is mostly focused on laboratory training, it is possible that Purdue may be finding it difficult to recruit specialist instructors with the expertise and/or availability required to address all objectives.

34


Accordingly, the evaluation team recommends that all further short-term training positions be advertised:

• Across the entire university community;

• Through specialist development recruitment agencies such as DevEx; and

• Through specialist job boards.

Priority should be given to qualified candidate with experience in developing countries. Recruitment of regional experts, drawn from countries with a functioning SPS system (such as India), should be considered.

4. Establish Kabul-Based Program Officer As discussed earlier, the lack of a Kabul-based program staff has seriously eroded the SPS Program’s capacity to develop, coordinate, and execute activities. The evaluation team recommends that the program engage an appropriately-qualified Afghan scientist to run daily program operations.22

Among other duties, the Program Officer should:

• Serve as program focal point in Afghanistan, from MAIL and/or other Afghan actors to USDA;

• Be a key member in program/training design;

• Coordinate with MAIL and other actors to ensure training is appropriate and prioritized;

• Arrange all program logistics, from teaching space, supplies, equipment, security, and transport;

• Conduct baseline surveys, pre and post training testing, and document indicators/deliverables;

• Maintain a database of graduates;

• Network and coordinate with other actors in the SPS/food safety system;

• Undergo specific specialist training in SPS as well as training-of-trainers; and

• Support all foreign specialists in their teaching assignments.

Once sufficiently trained, the Program Officer would be able to conduct introductory, general, or training-of-trainers in-house, leaving more advanced topics to external experts. Engagement of a local Program Officer would be the most cost-effective and effective approach, as the person would most likely be housed cost-free at MAIL and program expense would be limited to salary, basic computer equipment, and low-cost rental vehicle.

TRAINING AND OTHER OPPORTUNITIES TO EXPLORE The program is significantly under spent, though there is sufficient time to deliver valuable, cost effective programming within the remaining program time frame.

Explore viability of assisting Government of Afghanistan to meet IPPC standards

22 The evaluation team, with appreciable experience of the Kabul labor market, is confident that a well-qualified Afghan can be easily found to serve as Program Officer.

35


Afghanistan joined the International Plant Protection Convention (IPPC), located in Rome, on June 6, 2013. Afghanistan has to now comply with 34 measurable indicators. As part of its objective of developing capacity in regulatory systems, the SPS program could provide analysis and mentoring to facilitate Afghanistan in meeting indicators.

Develop MAIL capacity to implement HACCP within the SPS mechanism Hazard analysis and critical control points, or HACCP, is a systematic, preventive approach to ensuring SPS success, and it is applicable to all program objectives. While Purdue’s two-day food safety workshop touches upon the concept, it needs to be addressed more systematically.

The evaluation team recommends that Purdue help MAIL and MoPH think strategically and identify critical control points in the Afghanistan SPS system to justify trainings in SPS at other locations (such as with customs inspectors) besides those located on the campuses for MAIL and MoPH. Undoubtedly there are additional critical control points in the flow of food and fiber in and out of Afghanistan where Purdue could build the capacity of the SPS operators.

Also, by working with MAIL and MoPH, the three institutions could also think strategically about how to provide the resources necessary to equip laboratories, such as those located at the borders, with the proper equipment and supplies. It is beyond the scope of Purdue’s USDA-funded SPS program to equip these laboratories. Nevertheless, by working with MAIL and MoPH on their strategic plan for these laboratories, Purdue could help their colleagues in these two Afghan institutions to articulate the need/justification so as to acquire resources for these laboratories from their own government or from other donors.

Training Suggested by Others The evaluation team recommends the Purdue SPS program and USDA explore the viability and suitability presented in Findings – Requests for Training & Equipment:

A. Train Exporters in Visual Exports and Certification Procedures Provide exporters practical training in visual inspections that denote obvious problems. Teach them certification standards and modalities for food exports.

B. Train Kabul Food Inspectors Train the Ministry’s 13 food inspectors stationed in Kabul. Training would concentrate on the basics of food inspection and hygiene and the ability to take viable samples for inspection. Doubtless, MoPH would also make a similar request for its processed foods inspectors.

C. Train Inspectors at Kabul Airport and Border Zone Train customs inspectors at Kabul Airport and the four major land-entry points in appropriate SPS practice, food safety, inspection, and sampling techniques.

D. Establish Border-Point Laboratory in Kabul and Nangarhar Explore the viability of training and equipping a border post laboratory with basic equipment in Kabul and Nangarhar as per MAIL Acting Deputy Minister Amanuddin Haidary’s request.

36


ANNEXES

37


ANNEX I: EVALUATION STATEMENT OF WORK

Statement of Work (SOW) Afghanistan SPS Evaluation

1. BACKGROUND Through the Afghanistan SPS Program, Purdue University College of Agriculture and School of Veterinary Medicine works with the Afghan Ministry of Agriculture, Irrigation and Livestock (MAIL) and the provincial Directorates of Agriculture, Irrigation and Livestock (DAILs) to build capacity to implement plant health, food safety, and animal health activities. The goal is to have the capacity to certify the safety of agricultural products bound for export; monitor plant, animal, and food systems for pests, diseases, and food borne pathogens in storage or in transportation with the objective to report occurrence, outbreak, and spread of pests; inspect imported consignments at ports of entry that might harbor regulated pests and diseases; rapid and accurate pest, pathogen, and food borne disease diagnostic capabilities; and maintain accurate records and timely reporting mechanisms. The SPS Program provides Ministry personnel with SPS responsibility with a base level of knowledge, skills, ability, and authority to fulfill their roles and meet their responsibilities.

The long term goal of the SPS Program is to build MAIL and the DAILs capacity in implementing an SPS Program. The specific objectives are to:

1. Build capacity to implement regulatory systems for quarantine and surveillance, incorporating international standards for plant health, food safety and animal health into Afghanistan’s laws and regulations.

2. Improve laboratory diagnostic capabilities for plant health, food safety, and animal health.

3. Train MAIL and the DAILs staff in risk management for plant health, food safety, and animal health.

2. PURPOSE The external evaluation will determine the effectiveness of USDA’s SPS Program in Afghanistan. It will identify mid-term results and how they relate to the intended objectives and the likelihood of lasting effects after the SPS Program’s end, i.e., the degree of sustainability.

Results included in the evaluation report are anticipated to be used to help guide any mid-program changes needed to realign SPS Program activities to overall program goals and objectives.

3. REQUIREMENTS

3.1 EVALUATION The contractor shall answer the following questions and consider them in developing the evaluation’s methodology and approach:

3.2.1 Is the SPS Program achieving its specific objectives and to what extent are program indicators being met? The contractor shall consider to what extent program objectives lead towards the achievement of USDA’s Strategic Plan goals and Afghanistan’s assistance framework.

3.2.2 Is the SPS Program reaching targeted beneficiaries? If not, why not? If yes, what are the results?

38


3.2.3 What internal and/or external factors are positively or negatively affecting implementation? How can these factors be further developed or overcome, respectively?

3.2.4 How is the sustainability of the SPS Program being addressed? Are the beneficiaries being followed up with in order to track their progress?

3.2.5 The SPS Program was established to train staff via training-of-trainers and direct methods. What evidence is there that this is having a sustained effect? How can it be improved?

3.2.6 How has the SPS Program contributed to the creation and implementation of a functioning SPS system and what needs to happen to improve/expand that benefit?

3.2.7 How are the implementers working to identify participants? Are they in constant contact with the Ministry?

3.3 Prepare a Final Report of the Evaluation

3.4 Present findings in the Final Report to USDA staff in Washington, D.C.

39


ANNEX II: KEY DOCUMENTS REVIEWED

Title Date Agency AuthorAfghanistan – SPS Capacity Building Development final concept paper USDA/FAS/OCBD Afghanistan Sanitary/Phtosanitary Project, Quarterly Report Q2: April - June 2012 27-Jun-12 Purdue University naAfghanistan SPS Expression of Interest Purdue UniversityAfghanistan SPS Project Overview for Evaluation USDA/FASAfghanistanSPSOriginalProposal Purdue UniversityAmendment to the contract activities and budget 31-Oct-12 Purdue University Kevin T . McNamaraAnimal Health and Veterinary Public health ACT MAILAnimal health training application for Mr. Samadi Purdue University and Kevin T . McNamaraBergdoll AAEP SPS Trip Report 09-Oct-12 Purdue University Laura BergdollBudget revision correspondence Jul - Nov, 2012 Purdue University and Budget revision correspondence Jan & Feb, 2013 Purdue University and Concept Paper for supporting the transitioning of funding of Animal Health Services implemented through the General Directorate of Animal Health and Production and currently supported by the EC funded Animal Health Development Program (AHDP) 29-Mar-13

General Directorate of Animal Health and

ProductionContact list of MAIL Deputy Ministers and Directors MAIL Qais OriaKhailCooperative Agreement b/w Purdue and USDA/FAS 09-Sep-11 Purdue University and Partricia R. SheikhFood safety laboratory training application Purdue University and Kevin T . McNamaraMAIL Organogram MAILMAIL-Industry Collaboration and Education Training -- Info for Embassy Purdue University Kevin T . McNamaraMAIL-Industry Program Memo 11-Feb-13 Purdue University Kevin T . McNamaraMAIL-US Food Industry Mentorship Program Application form Purdue University and Kevin T . McNamara

MoU b/w MoPH and MAILMay, 2012

MoPH and MAILDr. Ahmad F.

Shadoul and Mr. Guindo Ousmaive

Oliver Santiago Trip Reports SPS Jun, 2012 Purdue University Haley OliverOverall SPS PMP Document 12-Feb-13 Purdue University

40


ANNEX II – KEY DOCUMENTS REVIEWED, CONT’D

Title Date Agency AuthorOliver Santiago Trip Reports SPS Jun, 2012 Purdue University Haley OliverOverall SPS PMP Document 12-Feb-13 Purdue UniversityPlant diagnostics training application Purdue University and Kevin T . McNamaraPotential information for embassy on Food safety laboratory techniques training Purdue University Kevin T . McNamaraPotential information for the embassy on Animal health training Purdue University Kevin T . McNamaraPotential information for the embassy on Plant Diagnostics training Purdue University Kevin T . McNamaraProject quarterly report (Jan - Mar, 2013) Apr, 2013 Purdue UniversityProject quarterly report (Jul - Sep, 2012) Oct, 2012 Purdue UniversityProject quarterly report (Oct - Dec, 2012) Jan, 2013 Purdue Universityrebudget part2 amendment request - 26-Nov-12 Purdue UniversitySPS Amended PMP 30-Jan-13 Purdue UniversitySPS bibliography Social Impact John DennisSPS Draft Work Plan 09-Apr-12 Purdue UniversitySPS project; Assessment of needs Dec, 2011 Purdue University Bruce HamakerSPS revised work plan 12-Jun-12 Purdue University Haley F. OliverSPS team assessment meetings Feb, 2012 Purdue University

SPS team final reportFeb, 2012

Purdue UniversityRick Foster, Haley

Oliver, and Ramesh Vemulapalli

SPS workshop final report 06-Apr-11 USDA/FAS/OCBD Jessica MaxwellSPSDiagnosticsTrainingReport Final Version May, 2012 Purdue UniversitySummary of the conducted visits/audits during March –May 2012 from five provincial VPH departments with their BIPs Involved persons (MF and NR) Mar, 2012

five provincial VPH departments

Mr. Amam

Trip Report Vemulapalli Mar, 2013 Purdue University Ramesh VemulapalliVemulapalli SPS Trip Report July, 2012 Purdue University Ramesh Vemulapalli

41


ANNEX III: PROGRAM EXPENDITURE RATE ANALYSIS Conducted by Evaluation Team based on SPS program quarterly reports.

As of Dec 31, the Program has met 52% of it's planned expenditure target

year

calend

ar mont

h

progra

m month

Progra

m monthly

expend

estim

ate*

running

total e

stimate

*

Progra

m monthly

expend

estim

ate*

running

total e

stimate

*

2011 Oct 1 57,184 3%2011 Nov 2 57,184 114,368 3% 5%2011 Dec 3 57,184 171,552 3% 8%2012 Jan 4 57,184 228,736 3% 11%2012 Feb 5 57,184 285,920 3% 13%2012 Mar 6 57,184 343,104 3% 16%2012 Apr 7 57,184 400,288 3% 18%2012 May 8 57,184 457,472 3% 21%2012 Jun 9 57,184 514,656 3% 24%2012 Jul 10 57,184 571,840 3% 26%2012 Aug 11 57,184 629,024 3% 29%2012 Sep 12 57,184 686,208 3% 32%2012 Oct 13 56,171 742,379 3% 34%2012 Nov 14 56,171 798,550 3% 37% Actual Spend % of target2012 Dec 15 56,171 854,721 3% 39% 446,042 52%2013 Jan 16 56,171 910,892 3% 42%2013 Feb 17 56,171 967,063 3% 44%2013 Mar 18 56,171 1,023,234 3% 47%2013 Apr 19 56,171 1,079,405 3% 50%2013 May 20 56,171 1,135,576 3% 52%2013 Jun 21 56,171 1,191,747 3% 55%2013 Jul 22 56,171 1,247,918 3% 57%2013 Aug 23 56,171 1,304,089 3% 60%2013 Sep 24 56,171 1,360,260 3% 63%2013 Oct 25 67,915 1,428,175 3% 66%2013 Nov 26 67,915 1,496,090 3% 69%2013 Dec 27 67,915 1,564,005 3% 72%2014 Jan 28 67,915 1,631,920 3% 75%2014 Feb 29 67,915 1,699,835 3% 78%2014 Mar 30 67,915 1,767,750 3% 81%2014 Apr 31 67,915 1,835,665 3% 84%2014 May 32 67,915 1,903,580 3% 88%2014 Jun 33 67,915 1,971,495 3% 91%2014 Jul 34 67,915 2,039,410 3% 94%2014 Aug 35 67,915 2,107,325 3% 97%2014 Sep 36 67,915 2,175,240 3% 100%

Note1 "Program Monthly Expend Estimate" is derived by taking Purdue's original budget estimate per year, dividing by 12 (months).

This is not a precise estimation per month, as it does not account for the variable program implementation cycle, but

2 As such, by Dec 31, 2012, the Program had expended 52% of plan

it does provide a rough guide to expenditure rates lacking other data.

42


ANNEX IV: LIST OF KEY PERSONS INTERVIEWED Select Key Informant Interviewees: not including survey participants.

NAME AGENCY DEPARTMENT POSITION

AFAZAL, Mohammad MAIL National Horticulture Program (NHLP) Livestock Extension Support Officer

AHMAD, Wali Aminee MOPH Nutrition Department Micronutrient Consultant

AHMAD-ZAI, Mir Ahmad MAIL Animal Health Lab, Darulaman Director; Animal Health

ALAM Khan Afghanistan Customs Department Customs Central Laboratory Manager

ALIAS, Mohammad Tabasom Raisin Processing Co Private Export Manager

ASHRAF, Mohammad Masroor Food Processing and Packaging Private Director

BABORI, Abdul Ghafoor MAIL Plant Protection & Quarantine Plant disease Director

BAHEAR, Waheddullah MAIL Central Veterinary Diagnostic and Research Laboratory (CVDRL) Lab Trainer

Commerce - Export Promt Min Commerce Export Promotion Agency CEO

DENNISON, Tania MAIL AHDP-MAIL-Darul Aman Program Manager

FAIZI, Zakria MAIL Plant Protection & Quarantine Change Management Specialist

FOSTER, Rick Purdue Purdue

GARCIA, Lloyd USDA USDA ex SPS embed FAS Kabul

GHURIANI, Abdul Ghani MAIL Deputy Minister Office Deputy Minister

HAIDARI, Mir Amanuddin MAIL Plant Protection & Quarantine Director

HAIDARY Mir Amanuddin MAIL Deputy Minister Office Acting Deputy Minister

HALEY, Oliver Purdue University

MANZOOR, Ahamad Season Honey Processing and Packaging Private Chief Executive Officer

MCNAMARA, Kevin Purdue Purdue

43


LIST OF KEY PERSONS INTERVIEWED, CONT’D NAME AGENCY DEPARTMENT POSITION

NABI, Ghulam MAIL Plant Protection & Quarantine Pesticides unit Head

NAZIRA, Rahman MAIL Home Economics MAIL - Home Economics Director

NEEK, Mohammad Kabul Dairy Union Private Production Manager

NOOR, Mohammad Ayubi Kabul University Veterinary Science Faculty Dean

NOOR, Rahman MAIL Veterinary Production Health and Quarantine Department Director

Obaidullah Reedy Shahbaza Fruits Farm Private Executive Manager

OMAR, Mohammad MAIL Central Veterinary Diagnostic and Research Laboratory (CVDRL) Virology Department

OSMANZAI, Mohammad Aziz MAIL Plant Protection & Quarantine Director

REHMAN, Shamsur Aga Khan Foundation NRM Lab National Manager

RONA, Dr MAIL

Public Health Laboratory CVDRL (Central Veterinary Diagnostic Laboratory) Lab Supervisor

RUDERT, Brian USDA-MAIL VEGA – Capacity Building and Change Management Program (CBCMP) VEGA - CoP

SCHAUWERS, Willy

Program Director

TILSWORTH, Robin USDA

Agriculture Minister Counsellor

VERMULAPALLI, Ramesh Purdue Purdue Trainer

ZARGHONA, Azizi MAIL Central Veterinary Diagnostic and Research Laboratory (CVDRL) PCR/Virology Department

ZIAY, Ghulam MAIL Central Veterinary Diagnostic and Research Laboratory (CVDRL) Lab Director

44


ANNEX V: DATA COLLECTION INSTRUMENTS A. KEY INFORMANT INTERVIEW QUESTIONNAIRE

Name__________________________________ Date ______________ Agency ________________________________ Survey Team _________________ 1. What is your understanding of the objectives of Purdue’s Sanitary Phytosanitary (SPS) program (Q-1)?

2. Is the SPS program meeting its objectives (Q-1)?

3. What is your understanding of who the beneficiaries of the SPS program should be (Q-2)? 4. Is the project reaching these beneficiaries (Q-2)?

5. How are the implementers working to identify participants? Are they in constant contact with the Ministry (Q-7)? 6. The project was established to train staff via train the trainer and direct methods. What evidence is there that this is

having a sustained effect? How can it be improved (Q-5)?

7. What are the essential elements of a successful SPS program (Q-6)?

8. What are the problems affecting the implementation of the SPS program (Q-3)?

9. What activities have worked well in achieving the implementation of the SPS Program (Q-3)? 10. How has the project contributed to the creation and implementation of a functioning SPS system and what needs to

happen to improve/expand the benefit (Q-6)?

11. How is SPS being institutionalized in your agency (Q-6)?

12. How is the sustainability of the project being addressed? Are the beneficiaries being followed up with in order to track their progress (Q-4)?

45


B. SURVEY

These questions were posed to 107 graduates of AASPS survey through email. Non-respondents were followed up with two other emails. If still unresponsive, trainees were called twice, when possible. In total, 33 responses were received.

SURVEY QUESTIONNAIRE

Survey DateSurveyorSurvey Location

The USDA (US Department of Agriculture) has hired Social Impact, to conduct a mid-term evaluation of the Afghanistan Sanitary-Phytosanitary Project under which you were trained. Please complete the questionnaire to help the USDA and Purdue University to know whether the training was useful. All information is confidential.

1 Profession2 Years of Professional Experience3 Location4 Gender5 Name of Course6 Were you briefed on the purpose of this course?7 How well was the course material organized?8 Describe the pace of instruction of the program?9 Did you have sufficient access to lab facilities for the course?

10 Did you have sufficient access to supplies and equipment for the course?11 How often do you pratice the skills learned in this course?12 What was the predominent language in the course?13 Did the language of instruction negatively affect your learning?14 Did you have difficulty reading the printed material because of language?15 How able are you to intregrate the training in your work?16 If you are not able to use this training, why not?17 Since your training, has your department increased its actitivites in Sanitary / Phytosanitary? (SPS)18 If yes, please explain19 How would you rate the quality of the training?20 What recommendations do you have to improve the course the next time?

46


U.S. Department of Agriculture Foreign Agricultural Service

1400 Independence Ave., S.W. Washington, DC 20250

Date post:	16-Oct-2020
Category:	Documents
Upload:	others
View:	7 times
Download:	0 times