30448
Beneficiary AssessmentAn Approach Described
Lawrence F. Salmen
Poverty and Social Policy DivisionTechnical DepartmentAfrica RegionThe World Bank
February 1992
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
0* 30448
Beneficiary AssessmentAn Approach Described
Lawrence F. Salmen
Poverty and Social Policy DivisionA Technical Department
Africa RegionThe World Bank
February 1992
The author would like to thank Misgana Amelga, Consultant, Poverty and Social Poverty Division for her
assistance in writing this paper. Cover Design and computer layout: Michael MatovinaI...... ..
Table of Contents
1. Introduction IRationale IBeneficiary Assessment Usage at the Bank 2
2. Involvement of Managers and Related Authorities 5
3. Assessment Design 7Understanding the Context 7Target Population and Sampling 7Selection of the Assessors 7Orientation for Assessors 8Preparation of Final Report 9
4. Methodology 11Direct Observation 11Conversational Interviews 11Participant Observation 12
5. Monitoring and Evaluating the Assessments 15
6. Impact of Beneficiary Assessments 17
7. Timing and Cost 19
8. Institutionalization at the Bank 21
9. Conclusion 23
Annex 1. Interview Themes 25
Annex 2. Conversational Interview Method 27
Annex 3. Participant Observation - Residence in a Community 29
References 31
1. Introduction
Beneficiary assessment is a tool formanagers who wish tion -and this informnation is intended to be of immc-to improve the quality of development operations. This diate use to managers and policymakers responsible foris an approach to information gathering which assesses improving people's lives.the value of an activity as it is perceived by its principalusers. The approach is qualitative in that it attempts to Rationalederive understanding from shared experience as well asobservation, and gives primacy to the centrality of the The key assumption underlying beneficiary asses>-other person's point of view. As the Bank and others ment is that the people for whom development isengaged in development activities seek to do their work intended, the beneficiaries, too often lack a voice loudbetter, one key indicator will need to be how the enough and clear enough to be heard by the managers o'ultimate customer, or intended beneficiary, assesses the development activities: the corollary to this. and iust asvalue of this work, project or policy, as it affects his or imponant, is that the managers, both in developin-her life. The illumination of how an intended benefi- countries and donor institutions, do not listen to theciary appreciates a planned or ongoing developmental beneficiaries as well as they might, for lack of trainingactivity is the primary objective. or inclination, or both. The need for intermediation is
clear. Sound beneficiary assessment, by encouragingBeneficiary assessment is a systematic inquiry into people to express their beliefs and values, leads to
people's values and behavior in relation to a planned or development which responds to, while it promotes, theongoing intervention for social and economic change. fullerparticipation of people in their own development.This method draws heavily from the tradition in social In this way, beneficiary assessment may be seen asscience known as "qualitative research...that funda- forging what is often a missing link in developmentmentally depends on watching people in their own work, introducing the socio-cultural dimension sys-territory and interacting with them in their own lan- tematically so that programs and policies are designedguage,ontheirownterms"(KirkandMiller, 1989). Yet and implemented for and by the people for whom the\beneficiary assessment also includes direct observa- are intended and thus are sustained long after the lasttion, incorporating simple counting, and is expressed in disbursement of funds is spent.quantitative terms. The ultimate goal of beneficiaryassessment is to reveal the meaning people give toparticular aspects of their lives so that development The need to gain a practicable understanding ofactivities may better enhance people's ability to im- people's values and behavior is essential to any effec-prove their own living conditions, as they see fit. This tive development work. The way to proceed and thedemands close rapport between the practitioner of this precise method to employ will vary according to theapproach, the beneficiary and the development man- nature of the development activity, the beneficiaries.ager. The beneficiary assessment approach is not the managers (both host-country and donor institution).intended to supplant the questionnaire survey but to and the phase of the activity. Nevertheless, there areprovide reliable qualitative, in-depth information on general practices which should be followed in all ben-the socio-cultural conditions of a beneficiary popula- eficiary assessments and which set the course for this
1
2
kind of leaming. However, deviations from this course development. One new area of Bank activity where
intheformofintensiveandprotractedstaysataanypoint beneficiary assessmcnt has shown itself to be a useful
along the way will be made to accommodate the needs tool is in Lhe design. implementation and evaluation of
of each particular program. Social Action Programs (SAPs).
Beneficiarv Assessment Usage at the Bank Beneficiarv assessment can ensure that SAPs aredemand dnven. The approach can be used in two of the
phases of the implementation of a social action
Thirty-one bank projects in twenty countries and in program:
all regions of the world where the Bank has lending
operations have used the bencficiary assessmentmethod. /. Project Selection
The following list divides Bank projects by seven
predominantly sectoral categories. The number of The Social Action Unit orsimilarentity of the Social
projects in each sector is in parenthesis. Action Program has the responsibility to screen and
select proposals for micro-projects. The unit staff
* Urban (9) should be trained in the beneficiary assessment ap-
* Agriculture (6) proach in order to:
* Population/Health (10)* Industry (2) * assist the implementing agencies to ensure that it
* Education (2) is used in proposal design. Consequently, the
* Energy (I) project should be responsive to community con-
* Social Action Program (1) cems and priorities. This facilitates the screening
While past assessments have been for project design and selection and reduces the chances of project
or one-shot evaluation purposes, many of the ongoing rejection.
(8) and planned (7) beneficiary assessments will beiterative leaming processes undertaken periodically * ensure properevaluation and screening of a project
throughout the lifetime of a project. The beneficiary proposal, particularly in terms of determining its
assessment approach has provided insights on the per- relevance to community needs.
spectives of intended beneficiaries which were found
useful by managers in both the design and implementa- 2. Monitoring and Evaluation
tion of activities in all sectors and regions of Bank
operations. In most cases, host-country managers ex- Beneficiary assessment plays an imponant role in
pressed interest in continuing to use the approach. The the SAP after the project selection phase. It enables the
Status Sheet, found on the next page, lists projects that implementing agency to monitorand address any change
have used beneficiary assessment and projects forwhich in beneficiaries' attitudes or unanticipated adjustments
assessments are planned for FY92. thatmaybenecessarytoeffectivelycompletetheproject.It is useful in monitoring the degree of people's partici-pation in the project. Regarding evaluation, the quali-
Social Action Programs tative techniques used in this approach can be used to
measure the success of the project in responding to the
communities' needs and in gauging their level of satis-
The beneficiary assessment approach has shown its faction with the project. Beneficiary assessments have
versatility by beingeffcctive in avariety of sectors, such been conducted on one SAP, in Bolivia, and have been
as education, health, industry, agriculture and urban planned for two others, in Zambia and Madagascar.
Beneficiary Assessment Status Sheet
Beneficiarn Assessment Projects Commenced to Date
Latin America Africa Asia EMENA
Bolivia Burkina Faso Kenya Thailand Turke%urban (1) population (fI l population (1), urban (2' education (I)rural (3)social fund (1) Burundi Zaire Philippines Tunisiaurban (2) rural (1) health (2), urban (1)Brazilurban (2) Lesotho Senegal Indiarural (I) health (1) industry (1) agriculture (1)*
population (1)° health (1)Ecuador Ethiopiaurban (I) health (I) Uganda
industry (1)Ghanapopulation (1)* Mali
education (I)Nigeriapopulation (I )* Malawi
energy (] )**Ongoing
Total 9 Total 14 Total 6 Total 2
Beneficiarv Assessments Plannedfor 1992
Latin America Africa Asia Total
Columbia (1) Mali (2) Zambia (I) Indonesia (1) Countries 20NGO evaluation pop./health social fund energy Sectors** 7energy Projects to date 31
Cameroon (1) Ongoing 9Madagascar (1) population l Planned 8food security/soc. fund
Central African Republic (1)health. education andagriculture
0 Includes social actionprograms as one sector.
3
II
2. Involvement of Manager-s and RelatedAuthorities
While the focus of beneficiary assessment is clearly on or sampling, and feel increased ownership for thethe beneficiaries of a development policy or activity, assessment process;the target is the decision making of the managersresponsible for that activity. While recognizing the 4. At the conclusion of the assessment. The finalvalue of continuous interaction between managers and report should first be presented to managementpersons responsibie for assessment, there are four times orally, subsequently in rough draft. The perspec-where the involvement of managers is crucial to the tive of management should be incorporated in thesuccess of beneficiary assessment: final text.
1. During conceptualization, so that the managerfeels "ownership" for the assessment; While the target of beneficiary assessment is man-
agement decision making, the goal is the improvement2. At the design stage so that issues in the interview of a project or policy. Thus, the persons conducting an
guide, population covered, and quality of inter- assessment should be of service to the managers but notviewers all meet with management approval; be subservient to them. Similarly, the focus on the
beneficiaries should not become an over-identification3. During progress review, midway through any with their immediate interests. As stated regarding
particular phase of field work (or at periodic participant observer evaluation, a precursor to benefi-intervals if it is a ruming assessment) to provide ciary assessment: "Although the participant observerfeedback on relevance and utility of initial find- evaluator must know management, project, and benefi-ings, assist with revision of interview guide and/ ciaries well, his allegiance is to none of the three, but
rather to what the project is meant to be"(Salmen, 1987).
5
iiI
I
I
3. Assessment Design
Understanding the Context suggest sample sizes greater than those needed forbeneficiarv assessment. Because of the in-depth quali-tative methodology employed in this approach, longA development program should be understood in light conversational interviews, often cornplemented by par-of its history, its architects - individual and institu- ticipant observation, can provide a great deal of un-tional - and the place and people for which it is derstanding from a relatively small number of benefi-intended. The first step in any beneficiary assessment ciaries. Aside from the key factor of the manager'sis familiarization with the context in which the program judgment, other parameters of the extent of coverageof concem is situated. This entails three primary of the beneficiary population will be determnined byactivities: the variability of the population of concem and thecomplexity of the program. Given the importance of* Reading all pertinent documentation, includ- the cultural variables of values and behavior, sam-ing appraisal reports, supervision reports, evalu- pling should generally be stratified according toations, and consultants' reports. A review of ethnicity, class or caste, income, and gender. Addi-
broader country economic analyses would also tional factors - such as climate, topography, andbe useful. landholding pattems - may be included as determi-nants of discrete population groups for the assessment
e Interviews with key persons involved with depending on the nature of the program. While thedeveloping, implementing, or evaluating the one-on-one conversational interviews are meant to beprogram in donor, host-country govemment quantified - and hence must be numerous enough toand other relevant (academic, consultant, NGO) be considered significant by decision-makers - fo-agencies. cus-group interviews and participant observation are
done primarily for illustration and contextual back-@ First-hand exploration of sites covered by pro- ground and need not conform to the same standards ofgram being assessed. Despite the preliminary representativity as do the individual interviews (seenature of this stage of the inquiry, the method to Box 1).
be employed follows the naturalistic one of Whilemostofthesubjectsofdiscussionsheldduringbeneficiary assessment itself. Disassociation beneficiary assessment will be the poor persons forfrom officials representing authority so as to whom a development intervention is intended, atten-avoid identification with them by beneficiaries tion must also be given to the perspective of the otheris advisable. Unstructured conversations with key actors involved, the program managers, the fieldrandomly-selected beneficiaries, actual or in- staff or extension workers, and the community leaders.tended, regarding planned orongoing interven- Where any of these groups is numerous (field staff suchtion will help in planning the more systematic as teachers, health workers, or farm extensionists),assessment to follow. representative sampling should be carried out along the
same lines as for beneficiaries and stratified accordingto key variables.
Target Population and Sampling
Selection of the Assessors
The degree of coverage of the beneficiary popula-tion needs to be sufficiently large to serve as the basis In beneficiary assessment, more than in most otherof a management decision. While statistical sampling realms of social science, the human being is the majorprocedures may serve as a general guide, these will instrument of learning. The innate human faculties of
7
8
Box 1 Ethiopia: A Combination of Interviews and Observation
A beneficiary assessment was done in Ethiopia as pan of a population and public health project coveringthe central provinces of the countrv. People in this area belonged to four primary ethnic groups (Oromo,Amhara, Karnbata, and Hadia). The assessment was conducted bv a combination of conversationalinterviews and participant observation. The interviewing was done with a representative sample drawn
from 10 villages in each of the four ethnic populations. The average village had about 50 households; fivehouseholds were interviewed from each village, doubling the number for the Oromo tribe which
comprises by far the largest group.
Panicipant observers spent 2-3 weeks living in three villages in each of the four ethnic areas to gainan in-depth understanding of the socio-cultural context as a complement to the household interviews.
intuition, perception, and empathy are all crucial to the in the community of beneficiaries, will require greater
successful execution of a beneficiary assessment. A experience and academic training, generally at the
capsuledescriptionofagoodbeneficiaryassessorwould graduate level, than the conversational interviewer.be a practical communicator. More broadly, a personlikely to conducta good beneficiary assessmentmustbesensitive, respectful, unobtrusive, engaging, open, have Orientation for Assessorsgood recall ability and writing skills, and, above all, bea good listener. Males and females should both beemployed so as to best appreciate the gender perspec- To conduct interviews with beneficiaries for optimaltive. In particular, it is important to remember that in results, the selected assessors will need specific orien-mosttraditional cultures persons are most apt to address tation and guidance. The orientation should providesensitive issues candidly with someone of the same clear guidelines as to the goals of the interviews and the
gender. The best assessors are familiar with the particu- role of the assessor. This can sometimes be difficult as
lar culture in which the assessment is to take place; it is social science researchers often think that they have
essential that the assessor have a sound conversational innate observational and interviewing skills as well as
ability in the language of the beneficiaries. educational training and, therefore, have little to learn.Being a good communicator and listener will not be
Given the difficulty of conducting a conversational sufficient. Performing systematic qualitative observa-interview with an unknown person ofLen of modest tions requires additional orientation and preparation.
means in such a way that their responses can be used toimprove their conditions, and of recording this conver- Beneficiary assessors should be given an orientation
sation in an orderly and intelligible manner, one may to strengthen their ability to:think it most difficult to find good beneficiary asses-sors. Experience demonstrates, however, that suffi- * write descriptively, and accurately;cient numbers of persons possessing the requisite skills * keep an organized record of field notes;
for this work exist in even the smallest developing * separate useful detail from trivia; andcountry. * avoid being influenced by any preconceived
notions of what responses should be.The pool of applicants is broadened when consider-
ing that communication skills are more important than Gender-sensitive orientation is essential to gain anany particular academic discipline. While direction of understanding of social, cultural and economic con-
an assessment should generally be in the hands of straints faced by men and women. This should be apart
someone with advanced university education, inter- of everyorientationsessionevenwhentheprojectisnotviewerswiththepropermixofskillsmaybehighschool dealing with issues directly related to gender. The
graduates - though university level is normally where particular objective is to promote fuller participation ofqualified interviewers are found. The person conduct- women in all phases of projects by designing develop-ing participant observation, involving total immersion ment interventions in response to women's felt needs.
9The orientation should sensitize assessors to the Reporting Finding
unique circumstances of the community they will beinterviewing. Assessors should be well informed as to The final report should be utilization focused. Wherthe general project goals and the context in which they the assessor sits down to write the report. the audienccmust apply their social science skills. This is still should be kept in mind. The audience will generally bcimportant even when the interviewer is a local person. the project manager who will use evaluation results frras there mav always be differences of an ethnic or class the project design, implementation or evaluation pro-nature. cess.
Preparation of Final Report To make an assessment report as readable as pos-sible, it helps to clearly address each major issue, either
Analysis preconceived or which arose during the assessment.This means that the description. analysis and interpre-
Assessors need to record the chronology of events. tation for each question is in one place in the report. Tovarious settings, processes and key issues that arise. be readable and understandable, the report should avoidSome informal analysis should occur as interviews are technical terminology to the degree possible. Assessorsheld and observations made. It is possible that the need to be very focused to sift through the cumbersomeassessor will get ideas about possible analysis as he/she amount of data thev have accumulated and select onl \interviews people. These ideas should be recorded in relevant inforrnation. There should be numbers. de-the assessor's notes for reference during the preparation scriptions and direct quotes to give precision, perspec-of the final report. tive and life to the information gathered.
Assessors' reports should start with brief executivesummaries that include findings and conclusions. Theproject manager or decision maker often has limitedtime to read documents, a shon, concise repon is apt tobe more persuasive than a lengthy, detailed report.
---- I.-
4. Methodology
Using qualitative meLhods in research permits the re- to provoke managers to change that process for thesearcher, orin this case the beneficiary assessor, to look better. The dominant research mode for this approachat issues in the round. The assessor can look behind the is qualitative, earmarked by a premium on listening tostructured responses obtained in a quantitative survey others on their own terms where they work and live; theand gain some perspective with which to balance the dominant style is one of creativity and flexibility, adapt-numerical data and understand the reasons for the ing the particular method of inquiry to the needs andresponses. Experience with projects in various coun- conditions of a given program and locale. But thetries has shown that qualitative research provides a true- predominance of the qualitative approach, which buildsto-life picture of project beneficiaries because the pre- on empathy between interviewer and beneficiary, shoulddetermined categories of quantitative methods are ab- not undermine the importance of representing findingssent. This does not mean that there is no systematiza- in numerical terms. Beneficiary assessment strives totion for data collection. Qualitative methods used in quantify qualitative information within a descriptive,beneficiary assessment must have a framework that explanatory context. This approach to developmentalallowsforquantificationofthedatagathered. Giventhe leaming has three principal methods which are dis-complexity of this task, the social scientist involved in cussed here.qualitative research has to be trained at least as rigor-ously as his quantitative counterpart. Direct Observation
In quantitative research, the validity of the results This is the simplest of techniques. It involves count-hingesonaccurateresearchinstrumentconstruction. In ing, noting behavior and expression, and otherwisequalitative methods, validity hinges on the skill and registering notable facets of a particular developmentalcompetence of the researcherorinquirer. Any misinter- situation. The potential for quantifying observed phe-pretation of information obtained through interviews or nomena is major and immediate (See Box 2).observations leads to erroneous or invalid data.
Quantitative research enables the inquirer to mea- Conversational Interviewssure the reactions of a large number of people to a finiteset of questions, making it easier to aggregate a large These are also know as qualitative interviews, thenumber of statistical data. Qualitative research draws basic tools of inquiry for the practitioner of beneficiaryfrom open-ended interview topics to generate more assessment. In well-guided, naturalistic interviewingdetailed in-depth data, but often represents a smaller people reveal their feelings, thoughts and beliefs aboutnumber of respondents. This approach is used in a particular issue. 'The fundamental principle of quali-beneficiary assessment because it provides tailored tative interviewing is to provide a framework withininformation that can assist project managers to design which respondents can express their own understand-and monitor development projects. The value of the ings in their own terms" (Patton, 1990). The conversa-assessment is in its usefulness in assisting decision tionalinterviewisstructuredaroundanumberofthemesmakers to make effective interventions. ortopics selected bythemanagers forwhom the assess-
ment is intended (see Annex I); these are issues gener-The methods employed in beneficiary assessment ally centering on the perspective of beneficiaries aboutare designed to reveal the true concems of the popula- which managers want additional information, or under-tion affccted by a development process in such a way as standing. Interviews may be conducted either one-on-
11
12
Box 2 Bolivia: A Case of Direct Observation
In La Paz. Boiivia. it appeared ihaL a slum upgrading project. which provided water. sewagc, and street
improvemcnts. scrved as an incentive tor home improvement. Roughly 70()' of the inhabitants had
borrowed mone to improvc their homes. To better undcrstand thc correlation between the project and
the home improvemcnts. threc observers wcre sent to three oLher ncighborhoods in the city which had
| roughly the same characteristics as the ncighborhoods being improved. The observers spent 2-3 days
counting the number of houses undergoing improvemcnts in the previous year and found an average of
12%Dc in the three communitics. This finding helped to buttress the case for ancillary development being
brought about by intrastructural improvements.
one or in groups (focus group interviews). The indi- could inhibit the respondent. The interviewer needs to
vidual interview allows for freer expression of certain recall what has been said during the interview and
issues which might be suppressed or distoned in the record it in writing shortly after each one or two inter-
presence of peers. and lends itsclf to quantification. views. In certain cultures, however, interviewers are
Focus group interviews allow for a coverage of more given greater credibilitv and considered more serious
persons in a given time period and may elicit cenain andprofessional,whentheyhavepensandnotebooksin
responses from the incentivc provided bv the presencc hand: some researchers report communities where tape
of peers which would otherwise not havc been ex- recorders are no obstacle to communication but may
pressed. The difficultv of precise atribution of re- lead to greater rappon as the beneficiaries hear their
sponses to individual persons in a focus group interview own voices played back to them, (much as the intrusion
may be a drawback for this method. of an instant-developing camera is overcome by the
sight of one's own photograph). As with all aspects of
beneficiary assessment, the key is adapting the means
The key to conducting a conversational interview is of inquiry to the culture of interest to elicit the most
establishing a good rapport between interviewer and cogent message (see Annex 2). For one beneficiary
respondent. Once trust is created in the interview assessment which relied exclusively on conversational
setting, information elicited should be valid. Good interviews see Box 3 .
rapport will generate unsolicitcd data which may be
more important than direct rcsponszs to the predeter-mined topics in the interview guide. Respect for the Participant Observation
beneficiary is basic to the establishment of rappon. The
timing of the interview, its duration and time of day,
should be dctermined by what is most convenient and Most of the methodology of beneficiary assessment
acceptable to the beneficiary. Generally, interviews could be subsumed under participant observation -
should not exceed forty-five minutes at any one time; to with the important caveat that it be directed toward
cover the interview guide adequately may require two managerial decision making in a development context.
orthreeconversations. Theseshonerinterviewsduring Participant observation as used here is the protracted
repeat visits show respect for the beneficiary's time and residence of an outsiderin acommunity of beneficiaries
allow increasing rapport to be established. It is impor- during which sufficient rappon is established and in-
tant that each interview be conducted so as to elicit volvement in their everyday activities occurs to allow
precise responses and, once sufficient rappon has been for the kind of representation of their living conditions
established, suggestions as to how the interviewee feels which will enhance the development potential of the
the situation discussed could bc improved. community occupied. Unlike participant observationin social anthropology, where residence in a commu-
nity being studied may last two or three years, the
Interviews should be conducted as naturallv as pos- panicipant observer in a beneficiary assessment gener-
sible, without the intrusion of any apparatus which ally resides in a communitv of beneficiaries for several
Box 3 Senegal: Interviewing Businessmen to Understand Private Sector Constraints
The purpose of this assessmcn was to gain an understanding of the constraints which impede tnc mrowthof Senegalese businesses. The assessment w\as carricd out on threc stall members of a loca graduatcschool of management. GESAG. A representative sample of businesses irom food processing. cxtfiie.>chemicals, papcr machines. construction among others was used.
Findings
* The businessmcn felt that the Govemment impeded business transactions bv creating bureaucraticexigencies and onerous labor regulations.
* Businessmen resent what they consider to be unfair competition from the informal sector.
* Liberalization measures imposed as part of the New Industrial Policv were seen b! manybusinessmen to have favored the informal over the formal sector.
* Generally. businessmen felt that those who formulate the policies and create and administer theregulations regarding industrial development have little understanding of and therefore areunresponsive to the needs and concems of the businessmen of Senegal.
Impact:
The assessment was able to meet its objective of conveying the concems of a representative group ofSenegalese businessmen. The findings are expected to be put to use bv the Bank and the Govemment asthey jointly act to develop the private sector of Senegal.
weeks to two or three months. The interest here is political context in which the beneficiary lives an('focusedonthesametopicswhichmakeupthe interview works (see Box 4). To best illustrate this contextualguide and other topics considered by managers to be dimension of the topics of interest, the participantrelevant to their concems but too sensitive or too observer generally conducts case studies on a smallpervasive to lend themselves to the visiting-interviewer number (usually 5 to 10) of beneficiary households whoformat. The emphasis in panicipant observation, are visited many times over the course of residence inhowever, is not only on the topics per se, but on how the community (for an elaboration of the participationthese issues are affected by the socio-cultural and observation method, see Annex 3).
Box 4 Ethiopia: Cultural Inhibitions to Prenatal Care
Participant observers in the beneficiary' assessment done in Ethiopia observed that pregnant womenalmost never visited health centers during their time of pregnancy. In-depth discussions held with menand women in the community in which they lived, revealed that the major reason for this low visitationrate lay in the cultural belief that it was considered weak and improper for women to admit to any painor discomfort. This information, which was new to the public health officials in Addis Ababa. wasconsidered usefui to help orient health education among rural communities.
i
i
5. Monitoring and Evaluating the Assessments
Given the non-standardized, tailor-made nature of ben- direct observation it will be possible to determine theeficiary assessment, and the sensitive nature of much of response of the community to the completed project.the information gleaned, it is necessary to monitor This input can be useful in identifying project strengthsbeneficiary assessments closely. Monitoring should be and weaknesses and in planning follow-up projects.done by persons not directly associated with the pro-gram being assessed, yet familiar with its essentialswho are well versed in the techniques of beneficiary Box 5 Bolivia: An Example of theassessment. Monitoring should always be done in Importance of Monitoringconjunction with program management.
The importance of the monitoring function canThe two key times formonitoring beneficiary assess- be ilustate in a Bankea uation o anments are roughly one-third through field work, when cultural project in Bolivia which was using theenough information has been collected to judge its ciaro a n approch The introutility and enough of the sample is stll to be interviewed ductory part of the assessment had been doneso that a revised assessment will be useful and, at the through correspondence. Approximatelyend, for the final report. A third possible monitoring through the Assessmenipoint would be during the final phase of data tabulation, was discovered that a number of the interview-to see that recorded information is properly categorized ers dicondced very rich, lenthy interviews
and aalyze. Toal tie formonioringshoul not ers had conducted very rich, lengthy interviewsand analyzed. Total time for monitoring should not with farmers which went back to their child-exceed three or four weeks, assuming two or three brief hood years prior to the revolution of 1952, butvisits during the execution of the assessment which hadnothingtodowiththeobjectivesof(See Box 5). the project. After an appropriate reorientation,
the interviews were streamlined, brought up toProper evaluation of the beneficiary assessment ex- date and directed to the evaluative goals of theercise is critical to determining the effectiveness of an atessmect t handintervention. Through conversational interviews and assessment at hand.
15
Ii
6. Impact of Beneficiary Assessments
Vcry often. new information and insights arc gencrated benefitted from these infrastructural improvements of-in the process of implementing a beneficiarv assess- ten failed to cxtend connections to renters. Authoritiesment. This was the case in onc projcct assessed for promptly established an ordinance requiring all land-which, from this new information, new regulations owners to makc connections for each and every house-were made in projcct communities, benefiting previ- hold resident, regardless of tenancy status. In Malawiously untargeted poor persons. In La Paz, Bolivia, as a a beneficiary assessment has helped make the electricresult of a slum upgrading project, infrastructural im- utility (ESCOM) more responsive to its consumers (seeprovements were made so that the residents had access Box 6). For an amplified description of a beneficiaryto water, sewagc and other facilities. Through partici- assessment on health in Lesotho and its impact (Seepant observation, it was learned that home owners who Box 7).
Box 6 Malawi: Promoting Electricity Usage in Rural Communities
The beneficiary assessment in Malawi had the objective of identifying socio-cultural factors as relatedto electric utility performance in rural setting. Three research teams, each comprised of a man and awoman, spent six weeks, one in each of three sites selected in collaboration with ESCOM (the local utility)on the basis of their geographical location and on their success as a rural electrification scheme.
Findings:
* Government officials' homes (policemen and primary school teachers) are not connected tonearby electric lines. Even though they are willing and able to pay for ongoing changes, they arereluctant to pay the high installment costs, given the possibility of relocation.* People are unaware of the real cost of electricity and overestimate it by as much as two to threetimes.
* Communication between ESCOM and its clients leaves much to be desired. While technicalmatters are commonly felt to be handled expeditiously, the delays in connecting houses,misunderstood readings and complicated electricity bills all give rise to distrust and alienationforn ESCOM on part of clients.
* While no precise data on employment generation is available, it seems that electricity clearly wasa boon to economic activity at the local level.
impact:
The major impact of this assessment is that it should help ESCOM be more consumer oriented than it haspreviously been, resulting in attracting more customers who may have the means to pay for the servicesbut who do not do so due to some misconceptions about utility services. It also offers the govemment ofMalawi the occasion to express its support of electricity as a development good which is becoming evermore important to the people of Malawi as a substitute for increasingly scarce fuel wood.
17
18
Box 7 Lesotho: The Impact of Beneficiary Assessment on Health Sector
In 1988, the Ministry of Health in Lesotho had the goal of expanding the provision of modem health
services to the country's population. The World Bank undertook to assist the Ministry towards the
achievement of health for all by the year 2000. It was understood that to achieve this goal it was necessary
to learn the attitudes, concems and customs of the people to design an appropriate health care system. An
in-depth qualitative analysis of individual and household behavior was conducted using beneficiary
assessment techniques. This approach was considered to be particularly relevant, given the sensitive
nature of the information needed, such as in family planning practices.
Three communities were selected. The participant observer method was chosen as the main
methodology. The participant observers were given two weeks of training and then were sent to live for
approximately two months in the communities where they would conduct interviews. Representative
samples of close to 50% were interviewed in three communities. Supplementary interviews were carried
out at clinics.
The following factors that would not have surfaced through traditional survey-techniques:
• The government village health worker (VHS) program failed because villagers did not use their
services, which were largely directed towards preventative health care. The VHS were not given
any curative remedies, even of a simple nature; this considerably lowered their status in the eyes
of the villagers. Traditional healers, on the other hand, were sought out by villagers because they
had curative remedies, herbs and such, for immediate use.
* Rural women often became pregnant because they did not have access to a steady supply of
contraception and having to travel long distances for supplies deterred usage. The husbands were
opposed to contraception, making it necessary for the women to hide their supplies.
* People were very dissatisfied with the quality of treatment and level of respect they received from
doctors and other health professionals in hospitals.
* The poor were excluded by fees beyond their means and the well-to-do were getting services at
what they considered to be low prices.
Impact of Assessment (project design):
* The government village health workers were provided with aspirin and other simple remedies to
facilitate interaction with the villagers;
* The traditional healers were brought into the national health system and given courses in basic
health;
* Contraceptives were made more accessible to women; and
* The need to categorize and charge patients according to socio-economic status was recognized.
7. Timing and Cost
The duration and cost of beneficiary assessments willvary markedly according to the nature of the programbeing assessed. From experience with World Banksupported projects, beneficiary assessments have gen-erally been conducted within a six-month period, fromdesign to the presentation of the final report. Thebreakdown of activities for the time period is generallyas follows:
Activity Months
Pre-test; training I
Survey; participant observation 2.5
Data tabulation I
Preparation of final report 1.5
Total 6
Most of the beneficiary assessments take an averageof six months and have cost under US$80,000. Bothcost and duration will increase when the assessment isdone, as is more often the case at present, in iterativefashion, at periodic intervals during (or throughout) thelife of the project.
19
ii
I
8. Institutionalization at the Bank
Considering the utility of beneficiary assessment to Thisapparentdichotomymaymoreusefullybemanagers, its low cost and timeliness, one might easily seen as a potential complementarily.question why, after roughly ten years of intensive use ofthis approach, it has been conducted in no more than * Bank staff do not have the training needed tothirty-one Bank projects. A number of reasons for the manage this approach.Bank's slow adoption of beneficiary assessment arepresented here: * People in the Bank generally do not understand
how beneficiary assessment is done or how* Beneficiary assessment appears to run counter little it costs; in some cases they do not know
to the Bank's culture: that it exists.
Beneficiary Assessment Bank * Many Bank staff and borrower-governmentpersonnel feel they know what needs to beknown regarding a development activity in-
inductive deductive cluding the people's perspective.bottom-up top-downsocio-cultural economic * Bank management provides few incentives orqualitative quantitive rewards forculturally-sensitive projectorpolicyprocess impact work.grounded abstractpractical theoretical Clearly none of this is immutable. Beneficiary as-
sessment can and should become a far more prevalentpractice in Bank operations.
21
I
Ii
9. Conclusion
By amplifying the voice of the people for whom devel- client populations. The ultimate test of the worth ofopment is intended, beneficiary assessment empowers beneficiary assessment is the creation and durability ofthesepeopletohelpthemselves. Byshowingdevelop- bridges of understanding which allow managers andment managers the value of improved communication beneficiaries to see that lasting development dependswith beneficiaries, this encourages the formation of on the integral involvement of both in support of goalsinstitutions more open and more responsive to their beyond themselves.
23
Ii
I
I
Annex 1. Interview Themes
Conversational interviews are created around themes of Energyimportance to managers. Below are interview themesused in beneficiary assessment in the areas of health, Topics for Discussion for Electricity Users:education and energy.
a. Process involved in obtaining utility serviceEducation b. Cost of acquiring connection and how paid for
c. Users of electricity in the household, business,a. Expectations from education institution and how often it is usedb. Schooling for girls d. Location where meters are installed and howc. Role of religion (obstacles to attendance) often and accurately they are readd. Quality of education received f. Problems encountered paying utility bill;
* degree of leaming perceived faimess of bills* utility of what is leamed g. Quality and effeciency of repair service
e. Distance from home to school h. Level of satisfaction with servicef. Physical aspects, maintenance, furnitures, i. Handling of complaints filed with the utility
general appearance . j. Attitude towards utility company; how serviceg. Language (for example, French, Arab, national) can be improvedh. Economic situation: is school seen as aid or
hinderance to productivityTopics For Discussion With Potential Users:
Health and Populationa. Interest in acquiring electricity service and how it
a. Health problems affecting family would be used; perceived advantages and disad-b. Perception of health service provided by vantages
(i) government b. Cost of being connected and how it compares with(ii) NGOs the cost of other energy sources(iii) traditional healers c. Perceived problems with paying bills
c. Degree and reason for attendance at health center d. Existence of user cooperatives, for example,d. Problems of village (agricultural production, farner's clubs or womens' clubs - potential as
water, education, etc.) vehicle for credit for electricitye. Family size e. Occurence of illegal connections and how usersf. Use of family planning methods and meaning of feel about it
the same f. View of utility company and services it provides;g. Suggestions for improving well being of family suggestions for improvements
25
i
I
IiI
Ii
i
III
I
Annex 2. Conversational Interview Method
What has been referred to here as a conversational or these works are finished. I wonder who will doqualitative interview is simply a way to broach a topic this later work..."of interest in a natural manner, as a part of a normalflow of discourse, such that the respondent will be at * "How is all of this infrastructure going to beease and induced to address the topic freely and with maintained?"candor. Sometimes direct questions, especially onsensitive topics, put people on the defensive and lead "These payments that you are being obliged toto responses which are considered in the respondent's make (for improvements) what is the idea be-best interests or what will please the interviewer. hind that?"Indirect interview methods are meant to elicit morevalid expressions of opinion, or of fact. * "Whose project is this, anyway?" (too direct
for early in conversation but may be a naturalIn a beneficiary assessment of an urban development comment after rapport has been established.)
(upgrading) project there was anumberof issues whichlent themselves to conversational interview techniques. (3) Suggestions for project improvements.Three of these issues are presented with the kind ofcomments which the interviewer may make to encour- * "Just as a visitor here, I'm impressed with allage a free-flowing dialogue: this project is doing to transform the neighbor-
hood. Idon'tknowhowitcouldbeanybetter?"(1) Adaptation of the project to the needs andpriorities of the beneficiaries. * "As in numberone (above) "Imagine you are in
charge of this project, what would you do to* "If you were designing a project to improve this improve it?"neighborhood what would you have done?"
"So, you're happy with what the project is* "Clearly, many major changes are being made doing; couldn't be better..."here (as a result of the project), the streets andwater (or public lighting, sewerage system, Once the respondent is provoked to give his or herschools, etc.) for instance. But then, I do not live opinions on a certain topic it is important that thehere as you do, so it's hard to tell what mally interviewer not appear to be overly inquisitive orjudg-matters mosL.." mental. The stance is one of being interested, support-
ive, respectful and somewhat innocent, or naive. Com-* "Certainly the designers of this project thought ments such as "really" or "that's the most interesting"
theywererespondingtoyourneeds,butofcourse or "well", I had no idea" are the kind of supportivethey are technicians and do not live here as you interjections which should lead to furtherelaborationofdo. Perhaps they missed something..." the issue being discussed.
(2) Degree to which people identify with the This is the people's story about a project that isproject. changing the place where they spend the days andnights oftheirlives. The interviewer wants to hear this* "This is a big project. The municipality is story and encourage its development. The respondent
making many improvements. Of course, a should be helped to understand that his or her opinionspoor neighborhood like this will need many will be transmitted to the project management so thatmore improvements and maintenance once the project may more closely respond to the beneficia-
ries' needs and priorities.
27
ii
Annex 3. Participant ObservationResidence in a Community
During the stay in a community, it is essential to get to the participant observer as a neighbor and, to varyingknow the key actors in the development program well: degrees, friend. At the level of the people, the relation-the beneficiaries in general, their leaders, formal and ship is far more personal than professional.informal; and the key administrators of the implement-ing agency. Attempt to keep the interrelationship with The participant observer should cultivate a few closeeach of these three major groups somewhat discrete. contacts from diverse, major segments of the popula-One's manner of relating is decreasingly professional tion. These should represent various key income groups,from program administrator to leader to regular benefi- political factions, owners and renters (where the latterciary. While all know the reason for one's stay in the are a significant part of population), etc. One shouldneighborhood, the official nature of that stay is to be never be over-identified with any one group, but bekept most apparent to the program personnel, with whom open and accessible to all: diplomacy at the neighbor-a certain degree of personal distance must be maintained hood level. Participant observers should participate into avoid bias, real or alleged, intentional or unconscious. major organizations and activities of the community suf-The leaders are also aware of the motive for one's pres- ficiently to be appreciated and identified as a participantence, especially at the outset; less so over time. The but not so much as to become over-committed. The aimpeople, however, while informed of one's reason for is to retain one's independence, yet demonstrateliving in the community, should primarily get to view involvement.
29
i
i
II
References
Agar, M. 1986. "Speaking of Ethnography." (Sage Miles,M.B.,Huberman,A.M. 1984.QualitativeAnaly-University Paper, Qualitative Research Methods sis: A Sourcebook of New Methods, Beverly Hills,Series, (Vol. 2,) Beverly Hills, CA: Sage. CA: Sage.
"From the Native's Point of View: On the Nature of Operations Evaluation Department (OED). 1985.Anthropological Understanding." In C. Geertz (Ed), Sustainability of Projects: First Review of Experi-Local Knowledge: Further Essays in Interpretive ence. World Bank.Anthropology. N.Y.: Basic books, 1983. Patton, Michael Quinn. 1980. Qualitative Evaluation
Chambers, R. 1988. Poverty in India: Concepts, Re- Methods. Beverly Hills, Califomia: Sage.search and Reality. Discussion Paper 241, Institute Rondinelli, D. 1983. Development Projects as Policyof Development Studies, Sussex, England. Experiments: An Adaptive Approach to Develop-
Glaser, B., and A. Strauss. 1967. A. The Discovery of ment Administration. Methuen, New York andGrounded Theory. Chicago: Aldine. London.
Guba,E.G.andLincoln,Y.S. 1981.EffectiveEvalua- Salmen, Lawrence F. 1987. Listen to the People:tion: Improving the Usefullness of Evaluation Re- Participant Observation Evaluation of Develop-sults Through Responsive and Naturalistic Ap- ment Projects, New York: Oxford University Press.proaches. San Francisco: Jossey bass. - 1989."Beneficiary Assessment Improving the De-
Havens, Lester, L. 1983. Participant Observation. New sign and Implementationof DevelopmentProjects".York: Jason Aronson. Evaluation Review. Vol. 13, No. 3.
Hirschman, A.O. 1985. Getting Ahead Collectively: - 1991. "Reducing Poverty", Public AdministrationGrassroots Experiences in Latin America. and Development, Vol 11.Pergamon, New York. Spradley, J. 1979. The Ethnographic Interview. New
Kirk, K. and M. L. Miller. 1989. Reliability and Validity York: Holt, Rhinehart & Winston,in Qualitative Research, "Sage University Series Participant Observation. New York: Holt, Rhinehart &on Qualitative Research Methods, Vol. I.Sage, Winston, 1980.Beverly Hills, Califomia. Stone, Linda and J. Gabriel Campbell. 1984. "The Use
Korten, K. and R. Klauss 1984. People-centered De- and Misuse of Surveys in Intemational Develop-velopment. Kumarian. West Hartford, Connecti- ment: An Experiment from Nepal." Human Orga-cut. nization 43, No. 1.
McCall, George J. and J.L. Simmons, eds. 1969. Issues Verspoor, A. 1989. Pathways to Change; Improvingin Participant Observation: A Text and Reader. the Quality of Education in Developing Countries.Reading, Mass: Addison-Wesley. World Bank, Washington D.C.
Merton,. Robert K., Marjorie Fiske, and Patricia L. Whyte, William F. 1984. Learning from the Field: AKendall. 1956. The Focused Interview. Glencoe, Guide from Experience. Beverly Hills, CA: Sage.IL: Free Press.
31
