,

DEVELOPMENT THROUGH PVOs and NGOs

ASSESSMENT DESIGN

Operations and Management Sector Working Group POUCDIFlElPOA

Design Team Members:

Albert H. Barclay, Jr. Michael Cacich

May 1993

Patricia L. Jordan, Assessment Manager Ronald D. Levin Richard Martin Carl E. Wisler

ATTACHMENT H

DEVELOPl\IENT TIlROUGH PVOs and NGOs

ASSESSMENT DESIGN

TABLE OF CONTENTS

ABBREVIATIONS AND ACRONYMS

I. INTRODUCTION

A. B. C.

Background Assessment Objectives Key Evaluation Questions

IT. DESIGN FR.Al\1EWORK

A. B. C. D. E. F. G. H. 1.

Study Approach DefInitions Conceptual Framework Assessment Indicators Selection of Activities Data Collection Data Analysis Side Studies Feasibility of Study Approach

ill. DISSEMINATION OF FINDINGS

A. B.

Expected Product Marketing/Utilization Plan

IV. WORK PLAN

A. B. C.

ANNEXES:

Expected Team Proftle Schedule Budget

A. Key Evaluation Constructs Matrix B. Data Analysis Options C. Using ArtifIcial Neural Networks to

Analyze A.LD. Projects D. Data Collection Protocols

Page 11

1 2 3

5 5 6 8

13 13 18 21 22

23 24

26 27 28

ABBREVIATIONS AND ACRONYMS

APR A.LD. AID/W

CDm CIDA LAC

NGO POA PVO

REDSO ROCAP USDH

Africa Bureau of A.LD. Agency for International Development Agency for International Development, Washington, D.C. offices

Center for Development Infonnation and Evaluation of A.LD. Canadian International Development Agency Latin America and Caribbean Bureau of A.LD.

Non-Governmental Organization Program and Operations Assessments Division, CDm Private Voluntary Organization

Regional Economic Development Support Office, A.LD. Regional Office for Central American Programs, A.LD. U.S. direct hire employee of A.LD.

- ii -

, ,

I. INTRODUCTION

A. Background

A.LD. long has used Private Voluntary Organizations (PVOs) for humanitarian and development assistance. The Agency ftrst turned to PVOs to handle short-term disaster relief and food distribution. Gradually, the use of PVOs was broadened to programs aimed at addressing the root causes of poverty and vulnerability to disasters. More recently, PVOs also have focused on developing the capacity of their indigenous partners in recipient countries, and A.I.D. has developed "umbrella PVO" projects to allow the transfer of funds and technical assistance for additional strengthening through PVOs to indigenous non­governmental organizations (NGOs). (See Section n.B., below, for the deftnitions of "PVO" and "NGO" as used in this study.)

In recent years, a number of donors, academics, development professionals, and members of the U.S. Congress have begun to recommend that PVOs and NGOs playa larger role in implementing development assistance and in advocacy roles, believing that use of grass-roots citizens' groups is the best approach to accomplish many development objectives. Moreover, the post-Cold War era has brought about a serious reappraisal of foreign assistance.

The dynamics of the A.LD./PVO partnership are influenced by changing world conditions and shifting development strategies. The ascendence of democracy, the shift toward market-based economies, the pew centrality of environmental concerns and the political shifts within Europe and the former Soviet Union have altered the context in which A.LD. and the PVO sector function. Groups like the Advisory Committee on Voluntary Foreign Aid, and studies such as the recent Canadian report, "A Time to Build Up New Forms of Cooperation Between NGOs and CIDA," suggest significant changes in the content and direction of the development process and the nature of the aid partnership with the PVO community.

As of September 30, 1992, there were 347 U.S. PVOs registered with A.I.D., not all of which were receiving USG support. The United States government, however, provided over I $1.2 billion in the form of grants, contracts, U.S. Government-owned excess property, ocean \ freight subsidies and P.L. 480-donated food in support of PVO-administered programs. Hundreds of NGOs also are registered with A.LD. Missions and received USG support from bilateral Mission funds.

In most A.LD. Missions, PVOs and NGOs are at work delivering disaster relief, handling food distribution, or implementing development assistance projects. They are sometimes called upon to test innovative development approaches through pilot activities. Moreover, A.LD. sometimes has directly funded PVOs and NGOs to implement assistance programs in countries where the host government was too corrupt, incompetent, or politically unacceptable for a traditional bilateral aid program. These organizations also have played a fundamental role in implementing A.LD. programs in more advanced countries where the Agency has a minimal in-country presence.

B. Assessment Objectives

In addition to the growing interest that the Congress and the new Administration have expressed in using PVOs to implement development assistance, A.I.D. has its own reasons for undertaking this assessment. First, A.I.D. has increasing concerns about the administrative costs of development assistance. If organizations such as PVOs and NGOs can manage more development assistance, fewer direct hire A.I.D. staff may be required. Second, the limited fmancial and administrative abilities of some recipient governments to implement development programs suggest the possibility that indigenous NGOs may be able to take more responsibility for implementing such programs. Third, A.I.D. needs to critically assess its past experience with PVOs and NGOs so that it can make sound judgments about the extent, circumstances, and ways it should work with them in the future.

Over the years, both A.I.D. and the PVO community have developed beliefs about their own comparative advantages as well as expectations or possible "myths" about each other. There is no doubt that PVOs and NGOs can be very effective in many areas and, at~ least for some kinds of activities, have a comparative advantage over other implementing mechanisms. Their self-reported advantages often include the following:

• They often have local ties and institutional structures in place which permit them to mobilize community participation.

• They work well at the grass-roots level and rely more on local talent than on expatriates.

• They often have developed expertise in specific areas of interest to A.I.D.

• They often are less expensive than other types of implementing entities.

A.I.D. 's relationship with PVOs and NGOs has not been without difficulties. A.I.D. evaluations have sometimes highlighted problems such as: (1) duplication of services; (2) proliferation of organizations, each with its own bureaucracy, staff and infrastructure; (3) small and spotty coverage; (4) adversarial relationships with government services and organizations; (5) religious and political agendas; (6) suspicions and rivalry among different PVOs and NGOs; (7) lack of policy leverage; (8) excessive dependence on donors; and (9) lack of adequate technical and professional staff.

For PVOs and NGOs, working within the A.I.D. system of rules and regulations -has not always been conducive to achieving successful results. Procurement, monitoring, and reporting requirements are often cited in this regard. A.I.D. 's increased reliance on cooperative agreements and contracts with tight controls has to some extent replaced the earlier use of grants. Even when grants are still used, the Agency no longer seems to recognize any differentiation in the degree to which A.I.D. 's regulatory controls are applied to the grant relationship. Some groups have become so disenchanted that they no longer pursue A.I.D. funding.

- 2 -

A preliminary survey of A.LD. documents has uncovered few assessments of A.I.D.'s use of PVOs and NGOs from the A.LD. manager's perspective, i.e., when it is in the A.LD. 's best management interests to choose a PVO or NGO to implement an A.LD. activity. Little empirical data have been gathered to support or refute commonly-held beliefs regarding the comparative advantages of PVOs as implementers of development activities or, for that matter, to assess the strengths or weaknesses of A.LD.'s implementation requirements.

With the above issues in mind, this assessment will provide some experience-based criteria for managers to suggest the circumstances in which PVOs and NGOs are most or least likely to be effective implementing mechanisms. Where the likelihood of effectiveness is strong, the study also will illuminate A.LD. management issues that are hindering effectiveness and thus require attention. At the same time, the study will try to show how efficient it is for A.LD. to work through these organizations under various circumstances, in terms of management burden.

No attempt will be made to address comprehensively all the issues in A.I.D. 's relationship with the PVO/NGO sector. The point of view taken will always be that of senior A.LD. management, faced with policy choices and seeking options for mechanisms to implement projects and programs. The assessment will not look for specific cases of success or failure, but will rather strive to identify, for the A.LD. manager's use, characteristics that suggest the likelihood of success or failure.

c. Key Evaluation Questions

This assessment will attempt to answer two major evaluation questions:

1. What PVO/NGO characteristics are most likely to lead to successful performance under specified country and project conditions? (NOTE: See Section IT.C., below, for the defInition of "successful performance. ")

2. What A.LD. management arrangements are most conducive to successful performance when PVOs/NGOs are the implementers?

The fIrst question is premised on the belief that if a sufficient variety of PVO/NGO experience is examined, it will be possible to identify those characteristics pertaining to choice of sector or project, the environment in which the activity takes place, and the nature of the PVO/NGO, which tend to augur for effectiveness. Conversely, it should be possible to predict where PVO/NGO efforts are likely to be less successful. The second question seeks to ascertain just how efficient A.LD. 's use of PVOs/NGOs is by examining the various management arrangements that have been tried.

Subsidiary chapters, or "side studies," of the assessment report will attempt to answer specific questions that have been raised in earlier evaluations or which may arise during the

- 3 -

course of key infonnant interviews, but which fall outside the principal study questions. Among others, the issue of sustainability will be addressed. In addition, the across-the-board application of A.I.D.'s financial accountability and procurement requirements, irrespective of the nature of the activity or the choice of funding instrument (contract, cooperative agreement or grant), is commonly cited as a serious impediment to effectiveness. This issue, too, will be addressed in a side study.

- 4 -

ll. DESIGN FRAMEWORK

A. Study Approach

The "Development through PVOs and NGOs" study is planned as the first of three related POA Operations and Management assessments. The second and third assessments will focus on development through consulting firms and through universities, respectively. v To the extent possible, this study will develop a methodology that can be used for the collection and analysis of data for all three studies, so that some comparisons can be drawn once all three studies are completed.

This study will be a limited field-based assessment, as defmed in Appendix B of CDIE's Procedures Guidelines. The main portion of the study will be devoted to collecting and analyzing data in order to answer the assessment's two key evaluation questions. In addition to the main study, a number of smaller "side studies" will be conducted to answer specific questions that may arise during the conduct of the study (see Section IT.H., Side Studies).

Much of the study's data will be collected from secondary sources, from an A1D/W­based review of existing documents. Qualitative data will be gathered from original sources, e.g., through interviews of key A.LD. and PVO/NGO informants using rapid appraisal data gathering methods. In addition, the assessment team is considering conducting a survey of PVO/NGO staff and A.LD. managers on their attitudes, knowledge and perceptions toward each other. Such information could prove valuable to A.LD. senior managers in separating "myth" from "reality" in A.LD.-PVO relations. Focus groups also may be used to formulate recommendations for solving specific problems identified in the study. Additional information on the conduct of data collection activities is provided in Section IT.F., Data Collection.

B. Deimitions

For the purposes of this study, PVOs and NGOs are defmed as non-profit, voluntary organizations that are operated by private, not public, management units. PVOs and NGOs work on behalf of a constituency (international or host country), from which they receive fmandal and/or in-kind support. PVOs and NGOs, in this study, do not include universities or private businesses operating on a not-for-profit basis. The approach of the majority of PVOs and NGOs is people-to-people, focusing on having an impact at the village and . community level. The term PVOINGO in this assessment refers to the universe of both U.S. and indigenous non-profit voluntary organizations. "PVO" refers specifically to U.S.-based non-profit voluntary organizations, while the term "NGO" is used to indicate indigenous organizations.

- 5 -

C. Conceptual Framework

The assessment will examine the relationship between perfonnance outcomes in a sample of PVO/NGO projects and four sets of factors, the independent variables, that are considered likely to influence those outcomes. The theoretical basis for the assessment is exploratory in nature and does not posit any particular explanation for project success or failure. Instead, data collection and analysis will search for patterns and relationships among the four groups of independent variables while treating project perfonnance measures as the dependent variables.

1. Project Perfonnance (Dependent Variables)

Project perfonnance will be measured in two dimensions, effectiveness and management efficiency. From an operations and management standpoint, these are the critical perfonnance standards to be met in choosing an implementing organization for an A.I.D.­fmanced project. High scores against both types of perfonnance standards would indicate that A.I.D. 's objectives have been met through reliance on a PVO/NGO implementer, and that A.I.D. 's resources have been used efficiently.

A.I.D. managers typically judge effectiveness based on: (a) whether a project achieved its specified outputs, and (b) evidence that the outputs led to accomplishment of the major objective or purpose of the project. These criteria are the basis for rating effectiveness in the sample of PVO/NGO projects that will be studied. Data provided in independent fmal evaluations will be the primary source for scoring projects on these two criteria.

Management efficiency criteria are not always explicit in the A.I.D. project design process, with the result that direct measures of this variable are rarely included in final evaluations. Designers may assume that an implementing PVO or NGO has adequate accounting procedures, uses its limited resources wisely to get the most from its staff, and can produce timely, accurate reports on the activities being funded by A.I.D. Because such assumptions often are not stated, they do not generate specific performance standards that independent evaluators can later use as benchmarks for measuring management efficiency.

The approach taken in this assessment treats management efficiency as a matter of degree, based on the absence or extent of certain documented implementation problems. Four categories of problems will be recorded, where applicable, based on data in the evaluations: (a) PVO/NGO internal management; (b) staffmg, including scarcity of technical skills; (c) fmancial management, such as funds flow and accounting for expenditures; and (d) A.I.D. management concerns, such as conflict over priorities and responsibilities for project management.

Each of these problems has the potential to slow down the pace of project implementation or to dilute the impact of A.I.D. resources provided to the implementing PVO or NGO. Conversely, where these problems are avoided, the probability of efficient

- 6 -

project implementation is assumed to be much higher. For this reason, the assessment will treat the absence or extent of such problems as a proxy for relative management efficiency in project perfonllance. Data analysis will include procedures for aggregating high or low problem scores to establish a rating for more and less efficient performance in the sample of evaluated projects. The individual performance variables are discussed in Section IT.D., below.

2. Other Factors andependent Variables)

Four groups of independent variables will be analyzed in the assessment in order to establish patterns that are associated with successful project performance. It is in the patterns discerned through the assessment that the two key evaluation questions can be addressed and answered. The four sets of independent variables are categorized as follows:

• Project characteristics.

• Country conditions.

• PVO/NGO characteristics; and

• Management arrangements.

The individual variables comprising these four categories of independent variables are discussed in the following section. These four categories provide a framework for analyzing the conditions - where, when and how - in which A.I.D. use of PVOs and NGOs leads to good performance at the project level. In this framework, combinations of these factors will be analyzed in a search for patterns that indicate the most favorable conditions for reliance on PVOs and NGOs as implementers of A.I.D. development objectives.

The number of variables is relatively large, because the literature on PVO/NGO development shows a very wide range of activities, operating conditions, project sizes, relationships with sponsoring entities in A.I.D., and organizational characteristics within the PVO/NGO community. The assessment is intended to reflect this diversity, rather than classifying PVO/NGO development activities into a small number of narrowly defined categories.

The assessment team does not expect to fmd single-factor causal relationships. Thus, there is no theoretical basis for imputing good (or bad) project performance solely to any factor, such as the size of the implementing PVO/NGO, the type of funding instrument (e.g., cooperative agreement), or the activity sector (e.g., child survival) in which a project operates. Management decisions in A.I.D. are not made this way, and the assessment's utility will depend on the degree to which it captures the multiple factors that are weighed in the actual decision-making process.

- 7 -

In the data coUection phase, indicators will be used to measure each variable in the four categories. During data analysis, it may be possible to aggregate the variables into a smaUer set. Thus, for example, in Section II.D. six indicators are described in the "project characteristics" category; when the data are analyzed, it may be appropriate to isolate a smaUer number of these indicators as the most critical factors affecting the probability of project success.

The conceptual framework assumes that patterns will emerge from data analysis indicating more or less promising scenarios for reliance on PVOs and NGOs. Those scenarios wiu be described in terms that closely match the real-world circumstances in which A.I.D. managers make resource aUocation decisions and set parameters for project implementation by PVOs and NGOs.

D. Assessment Indicators

As discussed in the previous section, dependent variables (project performance) and independent variables (explanatory factors) have been established for the study design. These variables were selected after consultation with a number of A.I.D. stakeholders who work with PVOs/NGOs. The selected independent variables represent factors which the stakeholders felt were most likely to be associated with project performance and for which there is accessible, reliable data. Each variable (or "construct") has a number of possible values ("attributes"). These are outlined below and in Annex A, the Key Evaluation Constructs matrix. Defmitions for each variable will be included in the instructions on the data collection coding sheets.

1. Dependent Variables

As discussed in the preceding section, the dependent variables, which measure project performance, are proxy measures for the effectiveness and management efficiency of the activity being studied.

EFFECTIVENESS

a. Accomplis/mzen! of activity outputs - This refers to the extent to which the activity's specified outputs (from the Logical Framework) were met. Outputs generally are quantitative. The possible values are: 76-100%; 26-75 %; 0-25 %; or unable to determine.

b. Achievement of activity purpose(s) - This variable refers to the extent to which the activity's specified purpose (from the Logical Framework) was met. This generally is a more subjective indicator and often uses success in meeting project outputs as one indicator. The attributes for this variable are: fully or mostly met; partiaUy met; not met; unable to determine.

- 8 -

~AGEMrnNTE~CrnNCY

c. PVOINGO internal management - This variable refers to whether the final evaluation documented problems with the PVO/NGO's internal management, e.g., over-involvement of Board members in daily operations or inefficient organizational structure. Attributes are: serious problems; moderate problems; minimal problems; none reported/can't detennine.

d. Staffing - This variable is concerned with the documented existence of personnel difficulties in the implementing PVO/NGO, such as inadequate technical skills, frequent staff turnover, insufficient personnel procedures. The values for this variable are: serious problems; moderate problems; minimal problems; none reported/can't detennine.

e. Financial management - This refers to documented problems in the flow of funds and accounting for expenditures, e.g., inadequate fmanciaI reporting and monitoring procedures, inability to become fmancially self­sufficient (if this is an objective), and inadequate planning and budgeting capabilities. This variable's attributes are: serious problems; moderate problems; minimal problems; none reported/can't detennine.

f. A.J.D. management concerns - This variable refers to documented on­going problems between the PVO/NGO implementing unit an~ the A.I.D. project managers over priorities and responsibilities for project management. The attributes for this variable are: serious problems; moderate problems; minimal problems; none reported/can't determine.

2. Inde.pendent Variables

The study's independent variables can be classified into four categories: those describing characteristics of the project being implemented; those describing country-related conditions; characteristics of the PVO/NGO; and management arrangements. The independent variables, grouped by category, are described below.

PROJECT CHARACTERISTICS

1. Activity sector - This refers to the general category of the project as classified by activity objective. The attributes of this variable include the following sectors: health/nutrition/child survival; family planning; education; small- and medium-scale enterprise; agriculture; environment/natural resources; community/grass roots development; capacity building; food aid; disaster relief; women in development; democracy and governance; and legal systems development.

- 9 -

)

2. Type of task - "Type of task" refers to the project's primary implementation activity. This variable includes the following values: policy dialogue (sector); policy dialogue (macro); technical assistance; training; service delivery; commodity delivery; and humanitarian relief/assistance.

3. A.J.D. objectives - This variable relates the activity's objective to the Mission's strategic objectives. There are three possible attributes: within Mission's primary objectives, within Mission's secondary objectives, and not within Mission objectives.

4. PVOINGO objectives - This refers to whether the objective of the activity is consistent with the primary development objective of the implementing PVO or NGO. The attributes are: within PVO/NGO objectives and not within PVO/NGO objectives.

5. Size of activity - This variable is defmed as total actual funding amount for the activity, expressed in dollars (i.e., A.J.D. funding plus PVO/NGO contributions, and including host government contributions, if any).

6. U. S. Congressional concern - This variable refers to whether the Congress has specially mandated activities in the same sector as the project or for a specific PVO. This can be measured by determining whether a relevant Congressional earmark or legislative report language exists.

COUNTRY CONDITIONS

7. A.J.D. in-country presence - This refers to the extent of A.I.D. staffmg in the host country. Possible values include: full Mission (more than 12 USDH staff); small Mission/AID Representative (3 to 11 USDH staff); and advanced developing country or no Mission (0 to 2 USDH staff).

8. Host government receptivity to PVOs - This variable refers to host government attitudes or practices toward PVO activities in the host country. Possible attributes are: receptive to PVOs; indifferent; and not receptive to PVOs.

9. Host government receptivity to NGOs - This variable refers to host government attitudes or practices toward NGO activities in the host country. Possible attributes are: receptive to NGOs; indifferent; and not receptive to NGOs.

10. Level of country development - This refers to the country's overall level of economic development. Attributes are: advanced developing country; less developed country; relatively least developed country; and transitional

- 10-

country (Le., Eastern European countries and those of the fonner Soviet Union).

PVOINGO CHARACTERISTICS

11. PVOINGO nation of origin - This variable refers to whether the PVO/NGO is based in the U.S. or is a host-country (indigenous) organization.

12. PVOINGO size - PVO/NGO size refers to the annual dollar level of the organization's funding.

13. PVOINGO internationalfocus - This variable is concerned with the degree to which the PVO/NGO is involved in international activities. The attribute is the percentage of international program funding (calculated as the annual dollar level of international program funding divided by the annual-dollar level of total organization funding).

14. PVOINGO program experience - This variable refers to the amount of experience in implementing development activities that the PVO/NGO has had in that sector prior to the initiation of the project being implemented. For PVOs, this refers to overseas experience; for NGOs, general development experience in the host country. The attributes include: extensive experience; moderate experience; or limited/no experience.

15. PVOINGO scope of operations - This refers to the extent of the PVO/NGO's specialization in one or more activity sector. The possible attributes are general (experience working in more than one sector) or specialized (experience working in only one sector).

16. Financial autonomy - This variable refers to the PVO/NGO's relative autonomy or fmancial independence, as measured by the proportion of total funding received from A.I.D. sources. The value for this variable is the percentage of total PVO/NGO funds received from A.I.D. sources.

17. PVO in-country presence - This variable is concerned with whether the U.S. PVO has an office established in the host country. The attributes for this variable are: established in-country presence; no established in-country presence; can't detennine.

MA1'\AGEMENT ARRANGEMENTS

18. Implementing mechanism - This variable is concerned with the instrument prescribed by A.I.D. governing implementation and accountability

- 11 -

arrangements. The attributes are: contract; cooperative agreement; and grant.

19. Management locus - This refers to the A.LD. management center for the activity. The values for this variable are: AJD/W - Central (Le., an activity managed from one of the A.LD. functional bureaus: Research & Development, Food & Humanitarian Assistance, and Private Enterprise); AJD/W - Regional (Le., an activity managed from one of the A.LD. geographic bureaus); Field - Regional Office (an activity managed from one of A.I.D. 's regional offices such as a REDSO or ROCAP); and Field­Mission (an activity managed by an A.LD. in-country Mission).

20. PVOINGO reporting requirements - This variable refers to the extent of A.LD. 's reporting requirements by the PVO/NGO for the activity. The attributes for this variable are: frequent/intensive reporting (defmed as quarterly or more often); limited reporting (defmed as semi-annual -or annual); infrequent or irregular reporting; end-of-grant reporting only; and no reporting requirements/can't determine.

21. A.I.D. monitoring - This refers to the intensity of A.I.D. 's project management activities, such as meetings with project staff, site visits and complying with A.LD. internal reporting requirements. The values for this variable are: frequent/intensive monitoring (defmed as quarterly or more often); limited monitoring (defmed as semi-annual or annual); infrequent or irregular monitoring; and no monitoring/can't determine.

22. Intermediary organization - This refers to whether A.LD. uses one PVO/NGO (or a PVO consortium) to program funds to a number of (usually smaller) PVOs/NGOs. A.I.D. often gives an intermediary organization an "umbrella grant," which the organization then sub-grants to other organizations. The values for this variable are: U.S.-based intermediary organization with U.S. branch affiliates; U.S.-based intermediary with local (indigenous) affiliates; and local intermediary organization with local affiliates.

23. Cost-sharing requirements - Cost-sharing refers to the resources that A.LD. requires the PVO/NGO to provide for the activity. The resources can be in the form of cash contributions and in-kind contributions, such as office space, supplies or staff time. The attribute for this variable is the percent of cost-sharing (cash and in-kind) required.

- 12 -

E. Selection of Activities

Activities to be studied will come from the pool of projects that have been directly implemented by a PVO/NGO. Bilateral projects will be studied only when there is at least one project component or activity where A.I.D. has provided funding directly to a PVO/NGO for project implementation. Bilateral projects where all project funding has been coursed through the host government will not be included in the study. For the purposes of this study, PVO/NGO involvement should be as project implementer rather than as project beneficiary .

The study will look at A.I.D. PVO/NGO projects that have started since 1980. Activities with and without fmal external evaluations on fIle at CDm will be included in the study; however, it is expected that projects with fmal external evaluations will be the principal data source. A sample of projects without external evaluations on fIle will be reviewed to determine if there are fmal evaluations on file elsewhere in the Agency, and/or to see if alternative data sources, such as mid-term evaluations and/or interviews with key informants, can be used to obtain the needed information.

Preliminary research of the CDm data base indicates that there are 1,034 projects, starting in 1980 or later, for which the terms "PVO" and/or "NGO" are used as primary descriptors. Thus, the potential "pool" of PVO/NGO project activities for inclusion in the study is 1,034. A number of these activities undoubtedly will not meet the study's definition of PVO/NGO activity, and will be excluded from further review. Of the larger pool of 1,034, there are 324 projects/subprojects which have fmal evaluations listed in the CDm data base. Further research will be required to determine how many of the listed evaluations are fmal external evaluations. Thus, the pool of PVO/NGO projects available for review is likely to be in the neighborhood of 700 - 800 projects, with 200 - 300 activities likely to have external evaluations.

F. Data Collection

I. Data Collection Methods. Sources and Instruments

The study's principal data collection methods will include AJD/W-based document review, AID/W -based interviews, and field-based rapid appraisal data gathering. In addition, the assessment team may use other techniques, such as a knowledge-attitudes-practices survey, to investigate special cases or issues. The data collection methods are detailed in the next section.

For the project-related variables, the A.I.D. management-related variables and all dependent variables, the primary data source will be final external evaluations of A.I.D. projects for which PVOs/NGOs acted as implementers. The primary data source for the country conditions variables will be A.I.D. planning documents, such as the Annual Budget

- 13 -

Submission, Congressional Presentation, and country strategy statements. A.I.D. central and Mission records will be used to obtain data on PVO/NGO characteristics. Other written data sources, such as midterm evaluations, project papers and grant agreements, will be used as needed. Interviews with A.I.D. staff at the central, regional and Mission levels, and with PVO/NGO staff, will be used to supplement written data collection or to clarify written information.

Data collection instruments will include a coding form for recording information collected from written sources, such as external evaluations and other background documents, and forms for recording information collected from interviews. Annex D contains a sample data collection form.

2. Data Collection Procedures

a. Pre-Test of Data Collection Methods

During the study design, a data collection exercise was conducted by four volunteers from the design team. Each person read several project documents and completed a coding form for each document. The resulting information then was analyzed to determine: (a) what types of data could be gathered from the different documents that were reviewed, and (b) the inter-rater coding reliability. As a result of this exercise, the design team has made a number of modifications to the primary data collection instrument and has narrowed its definition of acceptable study documents.

The assessment team plans an additional data collection exercise prior to implementation of full data collection activities. The purposes of this exercise are: (a) to determine if the list of identified variables is complete; (b) to assess the qUality of the secondary data sources; (c) to develop additional qualitative data sources; and (d) to pre-test and modify (if necessary) the data collection instruments. The exercise will involve AID/W -based document review and coding for PVO/NGO projects from one selected country, followed by limited fieldwork (10-14 days) in that country. The results of this data collection exercise may lead to some modifications in the assessment design prior to the conduct of the full data collection phase of the study.

To round out the pre-test phase of the study, the assessment team will undertake a short literature review on the management of A.I.D. PVO/NGO activities and will consult with other PVO/NGO experts in order to confirm the study approach and to incorporate other suggestions, such as additional variables, into the study design.

b. AIDIW Document Review

Initially, the study will focus on precisely determining the "universe" of A.I.D. PVO/NGO activities for review in the study. This will include the development of a definitive list of A.I.D. PVO/NGO activities started since 1980 and determining the number

- 14 -

of activities for which a final external evaluation has been completed (see Section IT.E., above). This list will be divided into two: one with activities that have fmal external evaluations, and one for PVO/NGO activities for which com does not have a final external evaluation on record.

For activities with fmal external evaluations, each evaluation will be reviewed and coded by an individual trained in the use of the coding form. (Coders will receive training and practice sessions on the proper coding of evaluations to ensure accurate capturing of data as well as inter-rater reliability.) Information not found in the external evaluations, such as country conditions and PVO/NGO characteristics, will be obtained from the appropriate source(s) and will be coded separately. Information from PVOs and NGOs will be coded individually, so that separate analyses can be conducted in the data analysis phase.

In addition to the activity-specific information, the document review may include gathering information on specific PVO/NGO management issues, which could be the focus of separate "side studies." Possible topics for side studies include sustainability of PVO/NGO activities or fmancial accountability requirements.

c. AID/W Interviews

Interviews with appropriate A.I.D. and/or PVO/NGO staff will be used as needed to complete data collection for each activity under review. Interviews with A.I.D. staff may be needed to collect information on such variables as: "A.I.D. objectives;" "PVO/NGO objectives;" "host country receptivity to PVOs (and/or NGOs);" "U.S. Congressional concern;" "PVO/NGO reporting requirements;" "intermediary organization;" the variables related to PVO/NGO characteristics; and the dependent (performance) variables (see Annex A, Key Evaluation Constructs). Interview forms will be developed so that the needed information can be easily captured.

d. Limited Fieldwork

Although most of the data collection will be pursued through AID/W -based document review and interviews, limited fieldwork will be conducted for two purposes: first, to verify data obtained through document review ("ground-truthing") and/or to fill in gaps in existing data; and second, to conduct mini-case studies in special situations where adequate A.I.D. documentation may not be available.

DATA VERIFICATION

The field-based data collection will be designed principally to verify the accuracy of the data drawn from the document review. The field team first will read all documentation available in Washington about PVO/NGO projects in the designated countries and, to the extent that the information is not already captured in coding forms prepared for the project, will document its judgments abqut the relative success of each project as well as specific

- 15 -

project indicators. Once in the field, the team will check the accuracy of the data already gathered. The fieldwork also will penn it infonnation to be gathered about project sustainability and will provide data for other side studies.

Fieldwork will include document reviews and rapid appraisal data gathering. Document review in the field will consist mainly of comparing local AID Mission fIle documents with the documentation found in the CDlE data base in Washington in terms of completeness. If evaluation documents are found on fIle in the field Mission that were not available in the Washington CDlE data base, the additional documents will be reviewed for two things. First, they will be checked for signs of selection bias. It will be important to detennine whether there is a consistent tendency for Missions to fail to submit certain kinds of evaluations - unfavorable ones, for example - to CDffi. Second, they will be reviewed for substance, to see if they contain important fmdings, insights, or recommendations to add to previously-compiled Washington infonnation.

The second field activity - rapid appraisal data gathering - will generate original data that will complement information obtained in document reviews. There are three reasons for gathering original data of this kind in the field. First, rapid appraisal data will make it possible to validate ("ground truth") the information obtained from evaluation documents. If independently-gathered information from the field corroborates conclusions drawn from the review of written PVO/NGO evaluations, then inferences and management implications can be drawn from the large sample of written evaluations with greater certainty. Second, rapid appraisal data will provide insight and detail not available in the written documentation, explaining why the events reported in a fonnal PVO/NGO evaluation really took place. Third, rapid appraisal data will add an important "human face" to the assessment, documenting subjective Perceptions and feelings about PVO/NGO activities in participants' own words.

The fieldwork will use two rapid appraisal techniques: focus groups and key informant interviews. Key informant interviews will be held with PVO/NGO executives and with AID executives. Interviews with PVO/NGO executives (normally PVO/NGO country directors and their senior staff) will deal with subjects such as their organizations' objectives and priorities; problems and advantages of working with A.I.D.; performance of their program; and their working relationships with other PVO/NGOs, the local government, intermediary organizations, and other donors. Interviews will follow a standard protocol, yielding comparable infonnation from different organizations and countries.

Key informant interviews with A.I.D. executives (normally AID Mission directors and other senior Mission staff) will deal with questions such as A.I.D. 's assessment of PVO performance; comparative advantages of PVOs for different kinds of tasks; management problems and advantages of working with PVO/NGOs; the political climate in the country for PVO/NGO activities; and the costs of PVO/NGO activities.

- 16 -

Focus groups will be organized with PVO/NGO project implementation staff, mainly local employees in charge of actually delivering services to clients. The focus groups will try to get information on the strengths and weaknesses of A.LD. support, including issues such as procurement and delivery of commodities; problems of contracting, hiring, and firing of staff; nature of A.LD. support for operating expenses; reasonableness of counterpart requirements; problems of A.LD. micromanagement or indifference; planning and evaluation practices; provision for eventual sustainability of services; paperwork and reporting requirements; and problems related to A.LD.'s political agenda or relationship with the host government. Focus groups also will follow a standard agenda, using a trained facilitator, to assure comparability across groups and countries.

CASE STUDIES

During the design phase, it became apparent that, in addition to the classic scenario where the PVO/NGO implements a project for a Mission or for AID/W with the cooperation of the host government, there are at least three special situations which fall outside this nonn and where accurate and complete data may not be readily available in Washington.

First, there are the cases of the Advanced Developing Country (AD C) programs. ADCs, usually A.I.D. "graduates," no longer need full Missions conducting hands-on programs. They continue to receive U.S. assistance for goodwill purposes, and the program is largely carried out by PVOs and NGOs. Second, to preclude the necessity of establishing large Missions and because the programs are expected to be short-tenn, A.LD. has designed its new programs in Eastern Europe to rely heavily on PVO/NGO activity. Third, in countries where it is not possible or desirable to work with the host country, A.LD. nevertheless may maintain a program of largely humanitarian assistance through PVOs and NGOs. Fieldwork in one example of each type of special situation will provide data regarding how effective and efficient the use of PVOs and NGOs is for these purposes. Fieldwork for the mini-case studies also will employ the data collection methods described above, document review and rapid appraisal (key infonnant interviews and focus groups).

The plan for limited fieldwork will be fmalized once the data collection process is underway, when there is more infonnation about where data gaps may exist. At this time, the plan is to visit six countries (including the pre-test country) - three each for data verification and for mini-case studies. Countries have been tentatively selected based on these criteria: existence of a broad range of PVO/NGO activities; geographic (regional) representation; and smaller countries or those less frequently visited by program assessment teams.

Tentative plans are for data verification field visits of 7-10 days each in: (a) Belize or Guatemala (LAC Region); (b) Bangladesh or South Pacific Regional (ASIA Region); and Mali or Tanzania (AFR Region). Belize has been tentatively selected for the pre-test of the data collection methods. The mini-case study visits will be about 14 days each, and may

- 17 -

include: (a) Mexico or Brazil (advanced developing countries); (b) Poland (new country program); and (c) South Africa or Haiti (use of PVOs as a surrogate for bilateral program).

G. Data Analysis

1. Overview

The appropriate form of data analysis depends upon at least three considerations: the nature of evaluation questions that are posed; the measurement levels of the variables of interest; and the number of cases for which data are available. Evaluation questions dictate the overall data analysis approach, while measurement levels and the number of cases may imply specific procedures.

Two key evaluation questions have been posed. These questions imply the need both to predict performance and to offer causal explanations for the observed performance in A.I.D. projects involving PVOs/NGOs.

Although prediction and causal explanation are related, they are not the same. One important consideration is that while causal explanations enable one to make predictions, it is often possible to make predictions even in the absence of causal explanations. A number of methodologies are available for making predictions, but causal explanation is much more elusive. Indeed, determining causality is often as much art as it is science. Annex B contains a summary of the approaches that were considered for analysis of the PVO/NGO data.

In this assessment, an effort will be made to go beyond prediction and search for evidence of causal explanation. Understanding of the strengths and weaknesses of PVOs/NGOs will be more complete if causal Iinks are evident. The main strategy will be to use several analysis methods. If different methods, having different strengths and weaknesses, point to similar causes, the support for causal statements will be bolstered. If the methods do not converge, then little can be said regarding causation.

After the data have been collected, they will be entered into a computer using appropriate software, such as an Ascn editor, spreadsheet, or specialized tool such as the Questionnaire Programming Language. The end product of this step will be a "zero-level" data base, containing the information as transcribed from code sheets and interview data sheets. Data analysis then will proceed in three roughly sequential stages: formulation of working data bases (including data editing and checking); exploratory analyses; and the main analysis to answer the evaluation questions.

- 18 -

2. Formulation of Working Data Bases

To verify the accuracy of the data, some spot-check comparisons of the data flIes will be made against the original coding sheets and interview forms. With minimal programming it also may be possible to do automatic checks of the zero-level data base for "outliers" (e.g. , a project with an extremely small dollar size) and logical inconsistencies (e.g., "level of country development" coded as advanced developing country and "in-country presence" coded as full Mission for the same case). Missing data also would be identified at this stage and imputed values determined as necessary for subsequent analyses. Finally, it will be necessary to create several different working data bases because of the requirements imposed by different analytical approaches. The definitions of variables, especially measurement levels, may vary across the working data bases.

3. Exploratory Analyses

To gain fanliliarity with the data, variables will be analyzed individually or in small groups. Elementary statistical analyses (e.g., computing central tendencies and bivariate measures of association) and graphical techniques (e.g., bar graphs and stem-and-leafplots) will be used to understand the data prior to more complex analysis. The exploratory analyses also will be used to check plausibility of assumptions that would be required for the main analysis (e.g., to check that variables have approximately a normal distribution or linearity of associations, if those were assumptions of certain analysis methods) . Another use of exploratory analysis might be to eliminate some variables from the main analysis.

4. Main Analysis

The purpose of the main analysis is to directly answer the evaluation questions. Each of the key questions aims to identify factors that have important effects on A.I.D. project performance. For the two evaluation questions, "What PVO/NGO characteristics are most likely to lead to successful performance under specified country and project conditions?" and "What A.I.D. management arrangements are most conducive to successful performance when PVOs/NGOs are the implementers?" the main data analysis framework can be structured as follows:

- 19 -

INDEPENDENT VARIABLES

DEPENDENT POLICY CONDITION QUESTION VARIABLES VARIABLES VARIABLES

I • Accomplishment of PVO/NGO characteristics (e.g., • Project variables (e.g., activity outputs PVO/NGO nation of origin, activity sector, type of

program experience, organization task)

• Achievement of size) activity purpose • Country variables (e.g.,

A.I.D. in-country presence, level of country development)

2 • PVO/NGO internal A.I.D. management arrangements • Project conditions management (e.g., particular implementing

mechanisms or cost sharing • Country conditions

• Staffing requirements)

• PVO/NGO characteristics

• Financial management ."

• A.I.D. management concerns

With the flrst question, the aim is thus to identify the characteristics of PVOs/NGOs that are successful under a variety of project and country conditions. The PVO/NGO characteristics are policy variables in the sense that A.I.D. can choose to support a class of PVOs/NGOs that possess certain characteristics.

Data analysis for the second question aims at identifying A.I.D. management arrangements that are associated with successful projects. An effort will be made to identify management arrangements that work under a wide range of circumstances (e.g., different countries, projects, and PVOs/NGOs), but the data may indicate that some customization will be advisable. That is, the preferred management arrangements may depend upon country, project, and PVO/NGO factors.

In brief, the approach to the evaluation questions is to develop ways to predict performance on a project from a set of variables. Performance can to some extent be "controlled" in the sense that A.I.D. can specify the values for policy variables. Although A.I.D. also has some control over the condition variables, it is to a lesser extent than for the policy variables.

The overall strategy is to strengthen the data analysis by using a mix of analysis methods to avoid drawing false conclusions because of any weaknesses inherent in a single analysis method. Even so, data analysis may show spurious or "surprise" results (results that were not anticipated and can not be easily explained). In that case, follow-up interviews, most likely at A.I.D. fleld Missions, will be undertaken in an attempt to explain the "surprise" results.

- 20 -

Two main fonns of analysis, logistic regression and artificial neural networks, will be used, along with some of the supplementary tools referred to under the preceding section on exploratory analyses. Depending upon the nature of the collected data, other data analysis options will be considered (see Annexes B and C).

H. Side Studies

During the assessment design process, a number of management concerns were raised which merit further investigation. Because data for these issues is not easily captured or analysed using the primary study approach, smaller "side studies" will be undertaken to explore and report on these concerns. As with the main study, these side studies will focus on the exploration of issues from the A.LD. manager's perspective. Data collection, analysis and reporting procedures for each side study will be developed prior to beginning the side study. The conclusions of each side study will be summarized in the main assessment report, and reported in full in an annex to the main report.

The following topics are under consideration for inclusion as side studies:

• PVOINGO SUSTAINABILITY: This side study would investigate to what extent A.I.D. funding, such as strengthening grants, has contributed to PVO/NGO organizational sustainability, and to what extent A.LD. requirements, such as cost sharing, have contributed to the sustainability of PVO/NGO projects or activities.

• COST EFFECTIVENESS: This side study will look at the feasibility of measuring benefits and of drawing comparisons among the three proposed operations and management studies (Le., development through PVOs/NGOs, consulting firms and universities). The output of this side study will be a design for a cost effectiveness study to serve as a companion to the three operations and management studies.

• A.LD. AD:MINISTRATIVE REQUIREl\1ENTS: This side study will address whether A.LD. 's administrative requirements (e.g., source/origin requirements; procurement procedures; salary level approvals; audit and fmancial reporting requirements; and competition requirements) are appropriate for PVO/NGO programming. Do they help or hinder project implementation? Are these requirements consistent with A.LD. policy toward PVOs/NGOs?

• NGO GROWTH: This side study will explore the A.LD. management implications of the growth of NGOs, both individually and collectively. What happens to NGOs as they grow? Is larger size a strength? How does this affect their implementation ability?

- 21 -

I. Feasibility of Study Approach

The design tearn perfonned a data collection exercise to detennine the overall feasibility of the study's approach to data gathering. The results of that exercise indicated that the overall approach is feasible given certain modifications to the study design, which have been incorporated into this document.

As a result of the data collection exercise, the design team detennined that fmal external project evaluations would be the primary data source for the following variables: project perfonnance (dependent variables); project characteristics; and A.LD. management variables. A.LD. planning documents will be used as the primary source of infonnation for the country conditions variables, and A.LD. PVO/NGO records will be used to obtain infonnation on the PVO/NGO characteristics.

The data collection exercise also resulted in modifications to the sample data collection protocol. Additional data collection tests, to ensure the quality of the written data and for further reliability and validity checking, will be conducted before a full data collection effort is begun. Any necessary modifications to the study design will be made at that time.

The design team believes that the overall approach to data analysis, using a combination of statistical and non-statistical techniques, is feasible. As indicated in Section ll. G. above, exploratory analyses will be made to detennine the suitability of the suggested approaches (logistic regression and artificial neural networks). Thus, the approach to analyzing the study data may be modified somewhat during this phase of the study.

- 22 -

m. DISSEMINATION OF FINDINGS

A. Expected Product

The main written product of the assessment will be a published report with a main text of approximately 40-50 pages, followed by a set of annexes. The contents of the report will be closely linked to the issues raised in this design paper. The basic outline of the main report will cover:

• Introduction and background, including A.LD. 's current policy toward PVOs and NGOs and historical trends in A.LD. 's use of PVOs/NGOs over the study period (1980-1993).

• Critical operations and management issues affecting A.LD. and the PVO/NGO sector: framework for evaluating effectiveness and management efficiency.

• Findings for evaluation questions 1 and 2: summary of evidence from the document review, interviews, and limited fieldwork.

• Summary of fmdings from side studies, including (provisional list): (a) sustainability of PVO/NGO projects; (b) impact of A.LD. administrative requirements on performance; and (c) pros and cons of central vs field programming of PVO/NGO activities.

• Principal conclusions and policy recommendations, including a discussion of the study's relation to the following studies on development through consulting firms and universities. The study fmdings and recommendations will be geared to: (a) senior Agency policy makers, and (b) operational managers with program and project responsibility.

Annexes will include the following: (l) a full explanation of the evaluation methodology; (2) a summary description of the sample used for document review; (3) details of data collection procedures and instruments; (4) a discussion of analytical techniques and the results obtained; (5) a summary of fmdings from each country visit for data verification and case study preparation; (6) full reports of the side studies; and (7) a bibliography of all documents reviewed for the assessment.

In addition to the main assessment report and related CDm publications, the assessment team will explore the potential for converting the assessment's data base and analytical methods into an automated information tool for A.LD. use. This system, called a "decision support system~ n could assist A.LD. managers who need to weigh a number of factors (which have been operationalized as independent variables in this assessment) when deciding when, where, and how to work with PVOs/NGOs in implementing future activities.

- 23 -

While the feasibility of developing this system is unknown at this point, such a system might have important ongoing utility for A.LD. managers at the regional and Mission levels. The system would be interactive, following the philosophy of "ask CDIE." Operating as a decision tool, the system, when requested, would take a proposed option with specific characteristics (e.g., in a given country and project sector, the selection of a small indigenous NGO to work under a cooperative agreement) and weigh it against the information recorded in the evaluation data base. The result would be an assessment of likely performance based on the specific characteristics selected. The A.LD. manager would take this information into account when making a decision. CDIE would continue to enlarge the data base in the system by adding new PVO/NGO evaluation data as reports from actual projects are acquired.

B. Marketing/Utilization Plan

Due to the fact that PVO/NGO programming is ubiquitous in A.LD. - encompassing most development sectors and taking place in most A.LD. field Missions and central bureaus - there is likely to be wide interest in the results of this study. Furthermore, the high profile of some PVOs/NGOs and their strong political interest and fmancial stake in A.LD. programming is bound to heighten interest in this assessment.

The assessment's target audience will include the following two groups of A.LD. managers:

• A.I.D. Senior Management. CDIE will encourage A.LD. 's top executive management and policy-making levels to use the assessment results to make policy decisions about PVOs/NGOs in Agency programming. Depending on the results of the assessment, senior management decisions might deal with topics such as: A.LD.'s management mechanisms for PVO/NGO programming (Mission vs. central programming, umbrella arrangements, endowments, etc.); possible special procurement regulations or provisions for A.LD.'s PVO/NGO activities; accountability standards for PVO/NGO programming; and possible sectoral or program priority areas (or exclusions) for A.I.D. support for PVOs/NGOs.

• A.I.D. Operational Levels. A.LD. program and project managers, both in field Missions and in functional! geographic bureaus, also will be informed of the assessment's fmdings and management implications. At the operational level, assessment guidelines hopefully will be used to focus PVO/NGO activities where they are most effective and to manage PVO/NGO activities more efficiently.

In addition to the target audience of A.LD. managers, other groups are likely to be interested in the study's fmdings, including the following:

- 24-

• PVOINGO Community and Constituents. PVOs/NGOs, their advocacy groups, and their supporters may find the assessment useful in identifying program areas and management styles in which PVO/NGO development activities in Third World settings have been especially successful. The study may help PVOs/NGOs to defme priority program areas and management styles in which their "comparative advantages" increase their chances of success, as well as identifying areas and styles they may want to avoid.

• U.S.G. Audit Units. The A.LD. Inspector General and the General Accounting Office may find the assessment to be informative, as it is expected to discuss areas in which A.LD. 's management of PVOs/NGOs can be improved. Other study areas that may be of interest include the impact of A.LD. 's control and accountability standards on PVO/NGO programming, and possible recommendations to improve fmandal management, control, reporting, and auditing of A.LD. 's PVO/NGO activities.

The main assessment report will be disseminated initially to A.LD. senior management. CDm will work with the other offices of the Policy Directorate to translate the assessment's "management implications" into a few specific, concrete policy recommendations, and will then present the policy recommendations to A.LD. senior management. To the extent possible, the policy recommendations will have specific action assignments and dates. The assessment report then will be disseminated to representatives of the other units identified above as stakeholders or decision makers. Depending on the level of interest, CDm also may offer briefmgs on the fmdings and management implications to a few of the units identified above.

If A.LD. senior managers agree on new Agency-wide policy guidance based on the assessment, CDIE may develop a special "A.LD. Package" consisting of the assessment report plus whatever new policy guidance is developed. This package would be distributed to Mission Directors and other units (procurement and contracting officers, sector technical officers, auditors and controllers, etc.), as appropriate. Distribution will depend on the nature of the policy guidance. CDm and A.I.D. senior management will decide jointly where responsibility lies for tracking the Agency's implementation of the policy recommendations.

In addition to the assessment report, the assessment team will produce several standard dissemination products, including an "A.LD. Evaluation Highlights" summarizing the report's fmdings and management implications, an oral presentation of results and management implications, and a succinct "Compendium" summary version. The "A.LD. Evaluation Highlights" will be sent to a standard A.LD. distribution list of about 2,300 readers, representing individuals and organizations with a professional interest in international development. The "Highlights" will invite interested readers to request copies of the full assessment report if they wish more detailed information.

- 25 -

IV. WORK PLAt"l

A. Expected Team Profile

The assessment team is expected to consist of the following individuals: two CDm staff (the Assessment Manager and one sector working group member); three methodology consultants (senior analyst, junior analyst, and methodology specialist); data coders; data entry personnel; and two field analysts.

The senior analyst will provide overall direction for the study's data collection and analysis stages. In the data collection stage, he/she will be responsible for the following: testing and revision of the data collection instruments; direction of the project sampling procedures; training of coders; development of data entry procedures; and development of specifications for the zero-level data base. In the data analysis stage, the senior analyst will be responsible for: verification and editing of the zero-level data base; missing data .' imputation; creation of scales/indexes; creation of working data bases; exploratory analyses; selection of analysis software; main data analysis; and technical report on data analysis. The estimated level of effort for the senior analyst is 60 person days.

The junior analyst will assist the senior analyst on such tasks as: testing the data collection instruments; development of data entry procedures; verification and editing of the zero-level data base; creation of working data bases; and exploratory analyses. The estimated level of effort for the junior analyst is 40 person days. The methodology specialist will provide technical guidance in strategies for data analysis using logistic regression and artificial neural network analysis. The estimated level of effort for this position is four person days.

Support personnel will include trained coders (approximately 100 person days of effort), who will extract and code data from the selected project documents (fmal evaluations). In addition, approximately five person days of data entry personnel assistance have been allotted for computer entry of the collected study data.

The assessment team also will include two field analysts, one each for the two sets of field visits (data verification and case studies). The field analysts will have experience in conducting field evaluations and will be familiar with A.LD. 's work with PVOs/NGOs, preferably in the countries each will visit. The estimated level of effort is 30 person days for the field verification analyst and 42 person days for the case study analyst.

- 26 -

B. Schedule

I TASK I ACTOR(S) I TARGET DATE I DESIGN:

Peer review of study design E:Binnendijk 4128/93

Design forwarded to external review panel CDIE:Smith 5/10/93 POA:Jordan

External review panel comments received CDIE:Smith 5/28/93 POA:Jordan

Revised design document completed, approved by COlE POA:JordanlLevin 6/4/93 management ETS:Wisler

DATA COLLECTION: .'

Identify, obtain documents for country pre-test POA:Jordan 5114/93 R&RS: Stallard

Literature review/stakeholder interviews POA:JordanlLevin 5/21193

Refine pre-test data collection instruments POA:Jordan/Levin 5121193 ETS:Wisler

Document review. coding for country pre-test POA:JordanlLevin 5/28/93 ETS:Wisler

"Universe" of PVO/NGO projects identified POA:JordanlLevin 5/28/93 R&RS:Stallard

Review coding, identify data gaps, prepare for pre-test POA:JordanlLevin 6/11/93 fieldwork ETS:Wisler

Projects selected for inclusion in study POA:Jordan/Levin 6/18/93 ETS:Wisler

Pre-test fieldwork (one country) POA:Levin 6/14 thru ETS:One contractor 6/25193

Project evaluations collected for review R&RS:Stallard 712193

Finalize data collection instruments POA:Jordan 7/16/93 ETS:Wisler ETS:Junior analyst

Training of coders ETS:Wisler 7/23/93 ETS:Junior analyst

Coding of evaluation documents ,ETS:Coders 7/26 thru 8/20/93

Collection of PVO-specific data (documents and POA:Levin 7/26 thru interviews) 8/20/93

- 27 -

I TASK I ACTOR(S) I TARGET DATE I DATA COLLECTION (con't):

Collection of country-specific data POA:Jordan 7/26 thru (documents and interviews) 8/20/93

Data entry ETS:Wisler 8123 thru ETS:Junior analyst 9/3/93 ETS:Data entry personnel

Field collection of data:

TEAM I: Data verification AID:One COlE 8/30 thru BangladeshiS. Pacific Regional ETS: One contractor 9/17/93 MalilTaozaoia

TEAM 2: Case studies Poland AID:One COlE 9/27 tnru MexicolBrazil ETS:One contractor 11112/93 S. AfricalHaiti

DATA ANALYSIS:

Formulation of working data bases POA:JordanlLevin 7/26 thru ETS:Wisler 8/20/93 ETS:Junior analyst

Exploratory analyses POA:JordanlLevin 917thru ETS:Wisler 9124/93 ETS:Junior analyst

Main analysis POA:JordanlLevin 9/27 thru ETS:Wisler 12123/93 ETS:Junior analyst

SYNTHESIS PAPER:

Drafting POA:JordanlLevin 12/27 thru ETS:Wisler, Barclay 1128/94

External review COlE: Smith 2/18/94 POA:Jordan

Publication PO~Jordan 6/17/94 CONW AL:Ebrahimi

c. Budget

The estimated budget for the data collection, data analysis and synthesis stages of the study is $170,000. This includes approximately $136,000 for personnel, $31,000 for travel costs, and $3,000 for equipment, supplies and other costs.

- 28 -

ANNEXA

Key Evaluation Constructs

POSSIBLE DATA COLLECTION CATEGORY CO~STRUCT ATIRIBUTES SOURCES METHOD

PROJECT a. ACCO~IPLISHMENT OF 76-100% Final external evaluation Document review PERFORMANCE ACTI\ "ITY OUTPUTS 26-75% CentrallregionallMission Interviews (Dependent 0-25% staff variables) Unable to determine

b. ACHIEVEMENT OF Fully or mostly met Final external evaluation Document review ACTI\'ITY PURPOSE(S) Partially met Central/regionallMission Interviews

Not met staff Unable to determine

c. PVOINGO INTERNAL Serious problems Final external evaluation Document review t-.L.o.NAGEMENT Moderate Central/regionallMission Interviews

Minimal staff None reported/can't determine .-

d. STAF~G Serious problems Final external evaluation Document review Moderate CentrallregionallMission Interviews Minimal staff None reported/can't determine

e. FINANCL-\L Serious problems Final external evaluation Document review MANAGEMENT Moderate CentrallregionallMission Interviews

Minimal staff None reported/can't determine

f. A.I.D. ~l"'NAGEMENT Serious problems Final external evaluation Document review CONCERNS Moderate CentrallregionallMission Interviews

Minimal staff None reported/can't determine

PROJECT I. ACTIVITY SECTOR Health/nutrition/child survival Final external evaluation Document review CHARACTER- Family planning Grant agreement ISI1CS Education

Small and medium enterprise Agriculture Environment/natural resources Community/grass roots dev't Capacity building Food aid Disaster relief Women in development Democracy and governance Legal systems development ..

2. TYPE OF TASK Policy dialogue - sector Final external evaluation Document review Policy dialogue - macro Technical assistance Training Service delivery Commodity delivery Humanitarian relief/assistance

3. A.l.D. OBJECTTVES Within Mission's primary Final external evaluation Document review objectives Project paper Interviews

Within Mission's secondary Country strategy statement objectives A.I.D. policy statement

Not within Mission objectives AID project staff

POSSIBLE DATA COLLECTION CATEGORY CONSTRUCT ATTRIBUTES SOURCES METHOD

PROJECT 4. PVO/NGO OBJECTIVES Within PVO/NGO objc!ctives Final external evaluation Document f'c!view CHARACTER- Not within PVO/NGO objectives CentrallMission records ISfICS (con't)

5. SIZE OF ACTIVITY Total actual funding amount for the Final external evaluation Documc!nt f'c!vic!w activity, expressed in dollars (i .e., Cc!ntral/regionallMission A.I.D. funding plus PVO/NGO records contributions, and including host government contributions, if any)

6. U.S. CONGRESSIONAL Congressional earmark Legislation Document f'c!\·iew CONCERN Legislative report language CentraUregionallMission Interviews

staff

COUNTRY 7. A.I.D. IN-COUNTRY Full Mission (more than 12 USDH) Annual Budget Submission Document f'c!view CONDITIONS PRESENCE Small Mission! AID Representative Congressional Presentation

(3 to II USDH) Advanced developing country/no Mission (0 to 2 USDH)

8. HOST GOVER..'lMENT Receptive to PVOs Country strategy statement Document f'c!view RECEPTIVm' TO PVOs Indifferent AlDIW or Mission staff .. Interviews

Not receptive to PVOs

9 . HOST GOVERNMENT Receptive to NGOs Country strategy statement Document f'c!\·iew RECEPTIVm' TO NGOs Indifferent AlDIW or Mission staff Interviews

Not receptive to NGOs

10. LEVEL OF COUNTRY A.I.D. graduate country Country strategy statement Document f'c!view DEVELOPMENT Advanced developing country AID/W reports

Less developed country Relatively least developed country

Transitional country

NGO II. PVO/NGO NATION OF U.S. PVO Final external evaluation Document f'c!view CHARACTER- ORIGIN Local NGO CentrallMission records Interviews ISl'ICS CentrallMission staff

12. PVO/NGO SIZE Annual dollar level of the organization's CentrallregionallMission records Document f'c!view funding PVO/NGO records Interviews

PVOINGO staff

13. PVO/NGO INTER- Percentage of international program CentrallregionallMission records Document review NATIONAL FOCUS funding (calculated as the annual dollar PVOINGO records Intervie';l.'s

level of international program funding PVOINGO staff divided by the annual dollar level of total organization funding)

14. PROGRAM EXPERIENCE Extensive CentrallregionallMission records Document review (PVO OVERSEAS Moderate PVO/NGO records Interviews EXPERIE~CE OR NGO Limited/none PVO/NGO staff GENERAL EXPERIENCE)

15. PVO/NGO SCOPE OF General (more than one sector) CentraI/regionallMission records Document review OPERATIONS Specialized (only one sector) PVOINGO records Interviews

PVOINGO staff

16. FINANClU AlITONOMY Percent of total PVOINGO funds CentrallMission records Document review received from A.I.D. sources

17. PVO IN-COIDI.7R.Y Established in-country presence CentrallMission records Document review PRESENCE No established in-country PVO/NGO records Intervie';l.·s

presence PVO/NGO staff Can't determine

- A-2-

POSSIBLE DATA COLLECTION CATEGORY CO!liSTRUCT ATTRIBUTES SOURCES METHOD

A.LD. 18. IMPLB{E~'TING Contract Final external evaluation Document ~view 1\tA.'I1AG EI\IENT MECHA!'IS~{ Cooperative agreement ARRANGE- Grant I\IENTS

19. MANAGE.\tEt-.'T LOCUS AID/Washington (central) Final external evaluation Document review AID/Washington (regional) Project paper Field (regional office) Field (Mission)

20. PVOINGO REPORTING Frequent/intensive reporting Final external evaluation Document review REQUlRE.\fENTS (quarterly or more) Grant agreement Interviews

Limited reporting (semi-annual Project paper or annual) CentrallregionallMission

Infrequent/irregular staff End-of-grant reporting only No reporting requirements/can't determine

21. A.I .D. ~fO~rrORING Frequent/intensive monitoring Final external evaluation Document ~view (quarterly or more) Grant agreement Interviews

Limited monitoring (semi-annual Project paper ,.

or annual) CentrallregionallMission Infrequent/irregular staff No monitoring/can't determine

22 . 1NTER.\fEDlARY U.S.-based wI U.S. branch Final external evaluation Document ~view ORGA.'lZATION affiliates Grant agreement Interviews

U.S.-based w/local affiliates Project paper Local w/ local affiliates CentrallregionallMission

staff

23. COST-SHARING Percent of cost-sharing (cash and in- Final external evaluation Document ~view REQUlRE.\fENTS kind) required

- A-3 -

ANNEXB

Data Analysis Options

The appropriate fonn of data analysis depends upon at least three considerations: the nature of evaluation questions that are posed; the measurement levels of the variables of interest; and the number of cases for which data is available. Evaluation questions dictate the overall data analysis approach, while particular measurement levels and the number of cases may imply specific procedures.

The analysis methods for dealing with issues of prediction and causal explanation often are similar, but with important distinctions in how they are used. Table B-1 shows the leading methodological candidates for analyzing data in the PVO/NGO study. The plan is to use some combination of methods.

To illustrate the main issues to be faced in the data analysis, consider the first question, about characteristics, when addressed as a question of prediction. Based upon data from available cases, it says "What kind of a PVO/NGO (Le., what PVO/NGO characteristics) is predictive of success for a given set of country and project conditions?" This is analogous to the situation facing a bettor who wants to pick the horse most likely to win a race. As long as he has a good way of predicting the winner - a good prediction tool - he doesn't care why some horses win and others don't. Likewise, for A.I.D. to pick a winner, it is not necessary to understand why a particular kind of PVO/NGO is the best bet - it is only necessary to have good evidence about past perfonnance for a variety of PVOs/NGOs operating under different conditions.

Viewing the issue as one of prediction, the aim of the assessment would be to provide Agency managers with a tool for deciding (predicting) which characteristics would be best for a particular circumstance. The tool would be applied when managers face the question of what kind of PVO/NGO to seek assistance from for a particular project in a particular country. Using linear regression as an example, the tool would be a regression equation produced by a regression analysis of the cases for which data were extracted from project documents. That is, a regression equation, with coefficients having numerical values based upon the analyzed cases, would be used to predict performance for a particular set of project and country conditions and for a set of PVOINGO characteristics.

For convenience, one set of PVO/NGO characteristics and project/country conditions will be called a candidate. A number of candidates, corresponding to different combinations of PVO/NGO characteristics, would be "evaluated" by the regression equation. The candidate corresponding to the highest performance would thus defme a class of PVOs/NGOs, those possessing similar characteristics and likely to achieve relatively high performance, from which to choose a particular PVO/NGO to implement the project. In sum, the prediction tool would be used to identify a class of PVOs/NGOs having the greatest likelihood, based on past performance, of successful performance on a future project.

Other methodological approaches (see Table B-1) could be used to make predictions in a similar way. For example, a neural network is trained, using the cases, to recognize patterns in the data. The prediction tool is then a trained network. A candidate is "evaluated" by the trained network which predicts performance.

With two or more tools in hand, the relative accuracy of predictions will be compared quantitatively. One way to do this will be to withhold approximately 10 percent of the available cases from the regression analysis, the network training, etc. Each prediction tool will then be tested against the reserved cases, for which performance is "known."

In the prediction mode outlined above, no effort is made to isolate the causal contributions of the various PVO/NGO characteristics nor, for that matter, the contributions of project and country conditions. As with the race horse analogy, the intent is simply to pick the likeliest winner in terms of performance.

By contrast, a causal explanation seeks a true representation of how the variables relate to one another. For example, in many commonly used approaches - regression is one illustration - a causal fmding is expressed as the amount of change in one variable that is produced by a change of one unit in another. To continue the race horse analogy, it is as if the race horse trainer wants to select a diet, training regimen, etc. that will increase the horse's chance of winning. She wants to know how much the likelihood of winning will be affected by increasing training time, decreasing the amount of carbohydrates in the diet, and so on.

In the PVO/NGO context, how much of a change in performance is associated with a unit of change in a policy variable or in a change of a project/country condition? For a variety of reasons, the methodological approaches in Table B-1 often do not work as well for addressing causal explanations as for making predictions.

Taking regression as an example again, the regression coefficients are key in causal explanations. They show how much of a change in performance is associated with a unit change in some other variable reflecting a PVO/NGO characteristic, or a project/country condition. However, unless the data used to produce the regression equation were generated by an experiment, the coefficients may be untrustworthy as indicators of causal relationships. Other indicators, such as variance accounted for by regression equations and partitioned connection weights in neural networks, likewise have limitations.

Another general point can be made about the fact that two evaluation questions have been posed. Conceivably, they could be combined from an analytical point of view. For example, all four categories of independent variable - PVO/NGO characteristics, project conditions, country conditions, and A.LD. management options - could be used to predict the accomplishment of activity outputs. Then the evaluation question might be "What policy options are most conducive to successful PVO/NGO performance under specified country and project conditions?"

- B-2-

At the present time, it seems inadvisable to collapse the two questions into one for three main reasons. First, it seems appropriate to consider A.LD. management options only as they relate to perfonnance on documented problem areas. A priori, it seems doubtful that the management options would show much association with accomplishment of activity outputs and activity purposes. Second, it seems appropriate to search for management options that can be applied regardless of PVO/NGO characteristics, or project and country conditions. As a fallback analysis, this limitation will be relaxed (see Type II analysis below). Third, while the predictive mode seems suitable for the fIrst question, the causal explanation mode seems more suitable for the second. Using results from the fIrst question, A.LD. managers will try to predict the class of PVOs/NGOs best suited to specifIed conditions. Using results from the second question, the managers may need to change A.LD. procedures. Causal explanation is a safer route to change than prediction. In sum, the use of two evaluation equations seems the simpler and more appropriate way to proceed. However, if preliminary analysis indicates otherwise, the two questions may be collapsed into one.

- B-3-

TABLE B-1

Data Analysis Options

TYPE OF ANALYSIS ANALYSIS FEATURES~ORTANT ADVANTAGES DISADVANTAGES METHOD METHOD TO STUDY FOR STUDY FOR STUDY

SlnllHlitnl Linllnr RIl~rllNNinn • RIl~rllHHinn cnllflicillnlH IINlld Itl chllruclllri7.1l Ihe Rdulivdy eUNY 111 expluin heCUIlNIl • CIt11Hul inlcrprclulil1nlll' Nlrenglh (impot1nnce) or Ihe independenl vArinhlcH . lI1uny penplellrll fillniliAr wilh Ihe rcgrcNsil1n cl1eflicicnls iN

• ASHumes linear relationship between dependenl mt!thod difficult hecause datH art! not variables and independent variables product!d by a trut! experiment

• Assumes dependent variable is continuous and • Requires a relatively large normally distrihuted numher of cases

• Single dependent variable • Missing data must be imputed • Requires relatively large number of cases or cases must be dropped from

analysis • Data may violate linearity

assumption

Discriminant Analysis • Linear combination of variables is used to predict ReNults are relatively easy to • Requires a relatively lnrge group membership (e.g., A.I.D. project will be in interpret number of cases successful or not-successful group) • Missing data must be imputed

• Method has fairly stringent assumptions, although or cases must be dropped from departures from some assumptions may still yield analysis usable results • Data may violate linearity

assumption

LogiNtic RegreHsion Similnr to regression nnAlysiN hut dependent vnrinhle iN • SimilArity tn linellr regressinn Snmc us fur lincnr regression mensured at the nominnl or ordinal level should mnke the method

relatively easy to explnin • Accommodation of nominal and

ordinal dependent variables avoids the need to make unfounded assumptions

• Fewer assumptions than discriminant analysis

Non-linear Similar to linear regression but the form of the • Similarity to linear regression • Same as for linear regression Regression regression can be intrinsically non-linear should make the method • No analytic solution to

relatively easy to explain equations so analysis may not

• Non-linear form of the converge regression.equation may be more realistic

- B-4 -

TYPE OF ANALYSIS ANALYSIS FEATURES IMPORTANT ADVANTAGES DISADVANTAGES METHOD METHOD .TO STUDY FOR STUDY FOR STUDY

Statistical (con 't) Exploratory A variety of statistical (and non-statistical) techniques Valuable for obtaining a Not suitable for obtaining an that can be used to understand individual variables and preliminary understanding of overall understanding of causal the relationships among small groups of variables variables relationships among study

variables -

Nun·Sllltislil-1I1 QUlllillllivc COlllpllrnlive • hlenlilieR the COllllilionH (illlierelllleni vllrillhleH) • AnlllyHiH rrlllhlceH II Hcl or • Any vnrillhle Ihlll is nol AnalYHis "clluslilly associllted" wilh dependent vllrillhles "canslll" cOllllilionH Ihlll are inherenlly hi-vIII ned IllUHI he

• Variables must be bi-valued generally easy to interpret (hut transformed to one or more hi-• Can be used with relatively small number of cases see disadvantage column) valued variables • Limited to 12 independent variables per analysis • Analysis is case-nrienled • Interprelation of some resulls • Limited to I dependent variable per analysis may be awkward because of

variable transformations

Artificial Neural • Suitable for all measurement levels and for non- • No compromises necessary in • Requires a relatively large Networks linear relationships among variables how variables are defined number of cases

• Cases are used to train the network to recognize • Can accommodate a relatively • Application of the method must patterns in the data large number of independent be adapted to evaluation use

variables • Likely to take more compuler • Can accommodate mUltiple time than other methods

dependent variables • Method is relatively

invulnerllhle tn misRing dllill • Analysis is case-oriented

Case-Based Reasoning • Suitable for all measurement levels and for non- • May be the most flexible option • Method is relatively new and linear relationships among variables in terms of how cases are not well understood

• Computer searches a case database to identify 8 case described • Procedures for drawing causal similar to "candidate" case • Relatively fast computations inferences not evident

• Case from database is adapted to apply to the "candidate" situation; inferences then drawn about the candidate case

- B-5 -

ANNEXC

Using Artificial Neural Networks to Analyze A.I.D. Projects

Artificial neural networks can be used to analyze patterns in data such as will be produced by the PVO/NGO assessment. An artificial neural network mimics certain aspects of brain structure, substituting "processing elements" in a computer for neurons in the brain's network. The figure below depicts a simple network with three layers of processing elements shown as circles. The elements are connected to one another in a way similar to how axons connect neurons. The strength of a connection between any two processing elements is expressed as a numerical value. A network's ability to recognize data patterns resides in the connection strengths.

Activity

Sector

Input Layer

PVO/NGO

Size

Country

Receptivity

Hidden Layer Output Layer

Achievement of

Project Purpose

Accomplishment of

Project ObJac:tlvas

lmernal Management

Weaknesses

The layer of input processing elements corresponds to independent variables in the A.I.D. project database, and the output layer corresponds to dependent variables. These two layers of elements are the means by which the network user communicates with the network-independent variables are presented to the input layer and, when the network is being used to solve a problem, the dependent variables are read from the output layer. (Although the figure above only shows three of each kind of variable, a network is subject only to software and hardware limitations.) A hidden layer of processing elements-there may be several such layers-is internal to the network and the user does not communicate

with it directly. One or more hidden layers gives a network improved capability to recognize patterns.

A network learns to recognize patterns in data by being presented with training cases, i.e. situations in which both the independent and dependent variables are known. In the PVO/NGO assessment, this means presenting a network with data (i.e., variables like those shown in the figure) from the projects for which information will be gathered. For a given set of values of the independent variables, network uses its connection strengths to "guess" the values of the dependent variables. During training the actual values of the dependent variables are known so the discrepancies between known and guessed values are used to adjust the connection strengths. Normally, the connection strengths change each time the network sees a new case. After having "seen" a number of cases, the network can correctly predict the dependent variables from knowing only the values of the independent variables. Such a network is said to be "trained." It can generalize to situations it has not seen before. A network must be trained before it can be put to use.

Once trained, the network becomes a management tool in that it can anticipate the results from a particular combination of PVO/NGO characteristics, country and project conditions. In practice, the network could be presented with a set of country and project conditions and it could then be used to explore the outcomes likely for a range of PVO/NGO characteristics.

An artificial neural network does not employ statistical assumptions and procedures although there are similarities between what the network does and non-linear regression analysis. The network analysis is best thought of as pattern matching-the computer learns to recognize patterns in the data and to generalize to new cases based upon patterns it saw during the training period .

•

- C-2-

/

PVO/NGO ASSESSMENT Code Sheet for Project Level Data - Draft 4/23/93

I. Project ID Code

2. Name of Principal PVOINGO

2A. Coding Confidence

2B. Primary Source

3. Name of Host Country

3A. Coding Confidence

3B. Primary Source

Coder Initials, ___ _ Date ___ _

Sure Not Sure

Document ID/pages

Sure Not Sure

Document ID/pages

. .................................................................................................................................. <14 ...................................................................................................... .

4. Established PVO In­country Presence

4A. Coding Confidence

4B. Primary Source

S. Activity Sector

SA. Coding Confidence

SB. Primary Source

6. End Date of Activity

6A. Coding Confidence

6B. Primal), Source

CODE-SH.V02

_In-country presence _No in-country presence

Unable to determine

Sure Not Sure

Document ID/pages

P = primary sector (only one).

~

S = secondary sector (may be more than one).

Health/nutrition/child survival _Population

Education Small- and medium-scale enterprise

_Agriculture Environment/natural resources

_Community/grass roots dev't

Sure Not Sure

Document ID/pages

(MID/Y)

Sure Not Sure

Document ID/pages

3

_Capacity building Food aid Disaster relief

_Women in development _Democracy and governance _Legal systems development _Other ( _________ --'

September 2, 1993

, . 7. Size of Activity

7 A. Coding Confidence

7B. Primary Source

S. Type of task

SA. Coding Confidence

SB. Primary Source

9. Accomplishment of Activity Output(s) (against primary objective)

9A. Coding Confidence

9B. Primary Source

10. Accomplishment of Activity Output(s) (against secondary objective)

lOA. Coding Confidence

lOB. Primary Source

CODE-5H.V02

$ ________ Total project cost

Sure Not Sure

Document ID/pages

P = primary type of task (only one). S = secondary type of task (may be more than one).

_Policy dialogue-sector _Policy dialogue-macro

Technical assistance _Training

_Service delivery _Commodity delivery

Humanitarian relief/assistance _Other l _______ -->

Sure Not Sure

Document ID/pages

Objective:

(Check 00> of Iho fol)ov.-iDp

_Project Incomplete _No evidence reported (project completed)

Percent objective achieved: 0-25 26-75 76-100

Nature of evidence:

Sure Not Sure

Document ID/pages

Objective:

(Check OlIO of Iho roUoo.-iDp

_Project incomplete _No evidence reported (completed project)

Percent objective achieved: 0-25 26-75 76-100 - -

Nature of evidence:

Sure Not Sure

Document ID/pages

4 September 2, 1993

I •

II. Achievement of Activity Purpose(s)

IIA. Coding Confidence

liB. Primary Source

12. PVOINGO Internal Management Weakness

12A. Coding Confidence

12B. Primary Source

13. Staffing Problems

I3A. Coding Confidence

13B. Primary Source

14. Financial Management Problems

I4A. Coding Confidence

14B. Primary Source

15. Management Conflict with A.I.D.

15A. Coding Confidence

15B. Primary Source

CODE-SH. VOl

Purpose(s):

(Cb:ck one "r Ih: follo..,inp

_Project incomplete _Fully or mostly met _Partially met

Not met Unable to determine

Nature of evidence:

Sure Not Sure

Document ID/pages

Serious Moderate Minimal

_None reported Unable to determine

Sure Not Sure

Document ID/pages

Serious Moderate Minimal

_None reported Unable to determine

Sure Not Sure

Document ID/pages

Serious Moderate Minimal

_None reported Unable to determine

Sure _Not Sure

Document ID/pages

Serious Moderate Minimal

_None reported Unable to determine

Sure Not Sure

Document ID/pages

s September 2, 1993

I ~

16. A.J.D. Objectives

16A. Coding Confidence

16B. Primary Source

17. PVOINGO Objectives

17 A. Coding Confidence

17B. PrinuJlY Source

18. Congressional Concern

18A. Coding Confidence

18B. Primary Source

19. hnplementing Mechanism

19A. Coding Confidence

19B. Primary Source

20. Management Locus

20A. Coding Confidence

20B. Primary Source

21. PVOINGO Reporting Requirements

21A. Coding Confidence

218. Primary Source

22. A.I.D. Monitoring

22A. Coding Confidence

22B. Primary Source

CODE-SH. VOl

_Within Mission's primary objectives _Within Mission's secondary objectives _Not within Mission's objectives

Sure Not Sure

Document ID/pages

_Within PVO/NGO objectives _Not within PVO/NGO objectives

Sure Not Sure

Document ID/pages

_Congressional earmark _Legislative report language _Neither

Sure Not Sure

Document ID/pages

Contract _Cooperative agreement

Grant

Sure Not Sure

Document ID/pages

A.I.D.lW-Central _A.I.D./W-Region

Sure Not Sure

Document ID/pages

_Field-Region Field-Mission

_No requirements or vague requirements _End of grant reporting only

Semi-annual or annual _Quarterly _More often than quarterly

Unable to detennine

Sure Not Sure

Document ID/pages

_No monitoring Less often than annual Semi-annual or annual

_Quarterly or more often Unable to detennine

Sure Not Sure

Document ID/pages

6 September 2, 1993

I _

23. Intennediary Organization

23A. Coding Confidence

23B. Primary Source

24. Cost-Sharing Requirements

24A. Coding Confidence

24B. Prima1}1 Source

25. Agent Responsible for Evaluation

25A. Coding Confidence

25B. Primary Source

26. Type of Measures Used in Evaluation

26A. Coding Confidence

26B. Primary Source

_U.S.-based wI U.S. branch affiliates U.S.-based wI local affiliates

_Local w/local affiliates No intermediaries involved

Sure Not Sure

Document ID/pages

__ Percent of project support (cash and in-kind) required of host country

Sure Not Sure

Document ID/pages

External evaluator NGO A.J.D. _Other(Doc ID---->

Sure Not Sure

Document ID/pages

_Measures developed for the evaluation _Measures available from activity or the country _Mixture of developed and available measures _Source of measures not explicitly cited

Sure Not Sure

Document ID/pages

Source Documents (Give title, date. and identifying alphanumeric codes)

1.

2.

3.

CODE-SH.V02 7 September 2. 1993

~ \ "

Name of Country

27. A.I.D. In-country Presence

27 A. Coding Confidence

27B. Primary Source

28. Host Government Receptivity to PVOs

28A. Coding Confidence

28B. Primary Source

29. Host Government Receptivity to NGOs

29A. Coding Confidence

29B. Primary Source

30. Level of Country Development

30A. Coding Confidence

30B. Primary Source

PVOINGO ASSESSMENT Code Sheet for Country Level Data - Draft 4/23/93

Coder Initials ___ _ Date ___ _

_Full mission _Small mission/A.LD. representative _No mission

Sure Not Sure

Document ID/pages

_Receptive Indifferent _Not receptive

Sure Not Sure

Document ID/pages

_Receptive Indifferent _Not receptive

Sure Not Sure

Document ID/pages

_A.LD. graduate _Advanced developing _Less developed

Sure Not Sure

Document ID/pages

_Relatively less developed _Least developed

Transitional

Sources (Cite title, date, and identifying alphanumeric codes for documents; cite name, title, and organization for interviews)

1.

2.

3.

CODE-SH.V02 8 September 2, 1993

• • t ..,

PVO\NGO ASSESSMENT Code Sheet for PVO\NGO Level Data - Draft 4/23/93

Coder Initials

Name ofNGO

31. PVOINGO Country of Origin

31A. Coding Confidence

31B. Primary Source

32. PVOINGO International Focus

32A. Coding Confidence

32B. Primary Source

33. NGO Size

33A. Coding Confidence

33B. Primary Source

-----Date -----

u.s. PVO Local NGO

Sure Not Sure

Document tD/pages

$ ________ Annual international funding

Sure Not Sure

Document ID/pages

$ _________ Organization's annual funding

Sure Not Sure

Document ID/pages

................................................................................................. : ......................................................................................................................................... . 34. PVOINGO Financial

Flexibility

34A. Coding Confidence

34B. Primary Source

___ Percent of funds from A.I.D.

Sure Not Sure

Document ID/pages

35. Program Experience Extensive Moderate Limited or none (PVO Overseas; NGO Host Country)

35A. Coding Confidence

35B. Primary Source

36. PVOINGO Scope of Operations

36A. Coding Confidence

36B. Primary Source

CODE-SH. VOl

Sure Not Sure

Document ID/pages

More than one sector _Only one sector

Sure Not Sure

Document ID/pages

9 September 2, 1993

Source Docwnents (Give title, date. and identifying alphanumeric codes)

I.

2.

3.

CODE-SH. VOl to September 2.1993