AOS Survey in BhutanDevesh Roy on behalf of Bhutan team

What is the annual outcome survey and why to do it? • IFAD has developed a standard methodology for impact measurement, these impact surveys are not

providing the type of results’ information that can allow Project Management Teams to take timely, corrective action during the course of project implementation.

• Such impact surveys are indeed primarily intended to document project impact at completion.

• To shift focus from impact documentation at completion to outcome measurement during project implementation, IFAD now encourages all projects Asia and the Pacific to survey annually a small sample of beneficiaries to:• measure more regularly the positive or negative changes/outcomes taking place at the household

level • provide early evidence of project success or failure• provide timely performance information so that corrective actions may be taken if required• assess targeting efficiency

• AOS is a simple Hh survey undertaken annually and cover a small sample of 200 households selected randomly.

• The survey conducted exclusively in villages targeted by the project or receiving project interventions, and will include both project beneficiaries and non-beneficiaries (the latter being used as a comparison group).

• Exercise expected to take no more than 3 months and can be implemented by project staff and extension officers, with or without external support.

Points in AOS

AOS survey design

• AOS should be conducted every year starting from the 2nd year of implementation.

• The survey to be implemented in conjunction with qualitative assessments that will complement the household-level information, providing information on “why” and “how” some outcomes were or were not achieved. • FGD • KII

• These two to be conducted after the surveys and data analysed• Suppose large groups of intended beneficiaries dissatisfied with the project, FGD and KII

explore why?

Steps in AOS (IFAD prescription)Activity/step Estimated Duration Comments Responsibility

Step 1 Fine-tuning of the standard survey questionnaire and preparation of interview guides for qualitative assessments (see TG 3 and 4)

1-3 days Can take longer if questions are added or modified from the standard template.

M&E Officer, with inputs from component coordinators and project coordinator.

Step 2 Sample selection 1-3 days Depending on availability of reliable lists. M&E Officer. Lists to be provided by project staff (villages) and local authorities (households).

Step 3 Training of enumerators and field testing of questionnaire and qualitative assessment guides

1-2 days The training should include a session on sampling.

M&E officer and external consultant if any.

Step 4 Logistical planning, preparation for data collection

1-2 days M&E Officer.

Step 5 Data collection 1-3 weeks Depends on availability of cars/motorbikes.

Enumerators and their supervisors

Step 6 Data entry 1-2 weeks If Excel file is used, data entry can only be conducted by one person at a time.

M&E Officer for supervision, Assistant for data entry.

Step 7 Data analysis 1 week M&E Officer with support from external consultant (if any).

Step 8 Report writing 1 week M&E officer, with inputs from Component Coordinators and Project Coordinator/Director.

Step 9 Communication and sharing ad hoc events Project coordinator, M&E and KM Officer.

Steps in AOSActivity/step Estimated Duration Comments Responsibility

Step 1 Fine-tuning of the standard

survey questionnaire and

preparation of interview

guides for qualitative

assessments (see TG 3 and

4)

1-3 days Can take longer if

questions are added or

modified from the

standard template.

M&E Officer, with inputs from

component coordinators and project

coordinator.

Step 2 Sample selection 1-3 days Depending on availability

of reliable lists.

M&E Officer. Lists to be provided by

project staff (villages) and local

authorities (households).

Step 3 Training of enumerators and

field testing of questionnaire

and qualitative assessment

guides

1-2 days The training should include

a session on sampling.

M&E officer and external consultant

if any.

Step 4 Logistical planning,

preparation for data

collection

1-2 days M&E Officer.

Steps in AOS: Continued Activity/step

Estimated

DurationComments Responsibility

Step 5 Data collection 1-3 weeks Depends on availability of

cars/motorbikes.

Enumerators and their supervisors

Step 6 Data entry 1-2 weeks If Excel file is used, data

entry can only be

conducted by one person at

a time.

M&E Officer for supervision, Assistant

for data entry.

Step 7 Data analysis 1 week M&E Officer with support from

external consultant (if any).

Step 8 Report writing 1 week M&E officer, with inputs from

Component Coordinators and Project

Coordinator/Director.

Step 9 Communication and sharing ad hoc events Project coordinator, M&E and KM

Officer.

Questionnaire developmentA – HOUSEHOLD IDENTIFICATION

A.1 Village ____________________________ A.1.a District or Province ___________________________

A.2 Name of respondent : ______________________________________________________________

A.2.a What is the gender (sex) of the respondent?A. Male

B. Female

A.3 Name of the head of household : ______________________________________________________________

A.4

What is the gender (sex) of the household head?A. Male ( Go to A.5)

B. Female

A.4.1. Is there a relative who takes care (financially) of your household’s needs?

A. Yes, a relative is taking care of us

B. No, no-one is helping us

A.5What is the size of your household/family (e.g. nb of persons living under the same roof)?

/_____/_____/ persons

A.6How many children (under 16 years), adults and elderly persons are there in your household/family?

/_____/ children

/_____/ adults

/_____/ elderly

A.7In terms of income group, in which of the following categories would you consider that your households belongs?

A. Well off

B. Average

C. Poor

D. Very poor

A. Yes

QUESTIONNAIRE A - FOR BENEFICIARIES

B – PARTICIPATION IN PROJECT ACTIVITIES

Since when is your household involved in project activities (year)? /_______________/ [year]

Over the past 12 months, have you (or a member of your household) participated in the following project activities?

[Adapt the response options below to the context of your project]

a - [Insert name of activity a] A. Yes B. No

b - [Insert name of activity b] A. Yes B. No

c - [Insert name of activity c] A. Yes B. No

Of the activities in which you have participated, which ones do you find were the most

useful ? [ Up to three responses only]/_____/ ; /_____/ ; /_____/

[Use same activity Code as above]

Of the activities in which you have participated, which ones do you find were the least

useful ? [ Up to three responses only]/_____/ ; /_____/ ; /_____/

[Use same activity Code as above]

How often do you have contact with project staff (extension workers; facilitators, etc.)?

A. Frequently (e.g. more than 2 times/year)

B. Occasionally (e.g. at least one time/year)

C. Rarely (e.g. less than one time/year)

A. Highly satisfied

B. Moderately satisfied

Stylized AOS questionnaire

• AOS stylized questionnaire.docx (see the elements of the questionnaire)

• A standard questionnaire. Questionnaire to be adapted to the specificity of project interventions.

• All sections of the questionnaire to be used as they are, or they can be modified according to the specific information needs of the project.

Questionnaire: Sampling

• The sampling frame (i.e. population from where the sample is selected) is the list of villages targeted by the project.

• Sample will be composed of 200 households (20 villages, 10 households per village).

• The selection of the sample done in two stages: (1) Selection of 20 villages; and (2) Selection of 10 households in each village.

• The same sampling frame will be used every year, but the sample selection will be re-done every year. Hence different villages anddifferent households surveyed each year.

AOS: Sampling Guidelines (Continued)

• Selection of villages• If villages similar- random sampling

• If villages dissimilar- stratified random sampling

• After selection of villages random selection of households

• Sampling interval = N/n (to the integer value)

• Random sample of villagers if list available (using sample interval)

• If the villagers list not available (transect line)

• Project beneficiaries and non-beneficiaries (how can they be ensured based on the sampling proposed?)

AOS survey in Bhutan

• AOS survey seems like a small activity but is quite involved• I wrote a long letter because I did not have time to write a short one

• The frequency can be demanding

• Well laid down rules- helpful always?

Market Access and Growth Intensification Project (MAGIP)

Some background on Bhutan and MAGIP• Six eastern Dzongkhags of Lhuentse, Tashiyangtse, Trashigang, Mongar,

Pemagatshel and Samdrup Jongkhar

• Implementing agencies – Ministry of Agriculture and Forestry in collaboration with district authorities and RAMCO (Combined ministry quite indicative) Separate Ministry of Finance and Economic Affairs also indicative

• Goal –Reduce poverty and improve food security

• Objective –enhance productivity of subsistence based farming where no road access, with road access promote cash crops and HVA, focus also on Alpine herders (incidentally not the poorest like pastoralists elsewhere since big herds)

• One Geog 3 product plan (OGTP)- what does it imply for MAGIP?

• Hierarchical geography like Nepal and young population like India

• Almost 90% rural population raises livestock in Bhutan

Characteristics of the targeted areas

• Six agro-ecological zones • Alpine (3600-4600 meters) – mainly summer pastures for yak herding

• Cool temperate (2600-3600 meters) –transhumant yak and sheep farming

• Warm temperate (1800-2600 meters)-migratory cattle herders with the cultivation of rice, wheat, potato and other vegetables

• Dry subtropical (1200-1800 meters)- comparatively warm with moderate rainfall- major crops rice and maize followed by wheat and mustard

• Humid subtropical (600-1200 meters)- high rainfall and temperature, cattle reared in sedentary system – rice, maize, mustard and citrus fruits

• Wet subtropical (150-600 meters)-very high rainfall, rice grown as the main crop in summer, maize and wheat in winter depending on irrigation

• Barring Alpine and Cool temperate others suitable for sustainable agricultural and livestock production.• Yet lack of infrastructure and market access have stymied progress in eastern region

Farmer differentiation by type in MAGIP

Background for AOS survey

• The improvements in reducing human deprivation in Bhutan have come largely from reducing malnutrition and improving access to safe drinking water.• Both the proportions of population without access to safe drinking water or in

malnourished category are below 10 percent (a remarkable achievement)

• Gender equality is built into Bhutanese society (matrilineal system of inheritance) and governance (yet disparities remain in education and employment-possibly a little late start)

• The RGOB does mainstream gender in its development plans

Facets about MAGIP

• MAGIP is in essence a targeted intervention (set of targeted interventions)

• MAGIP focuses on a small set of commodities and sub-sectors to maximize impact and support the emergence of rural market economy (in line with OGTP initiative)

• Impact is enshrined in the project

• MAGIP has new system of extension (Farmer Field Schools) and customized to clients like subsistence farmers, farmers with market access and Alpine hereers (losing ground with issues like fodder scarcity, changing consumers taste)

• MAGIP adopts a value chain approach

Elements in MAGIP• Agricultural development (Extension (FFS), fruit trees related

extension for better nutrition, mitigate post harvest losses (e.g. drying storage facilities), backyard poultry development, community forestymanagement, Pilot Agro Tourism Initiative

• Agricultural Intensification and support to market access (targeting farmers in 37 gewogs who have marketable surplus) (OGTP –vegetables, potato, dairy and citrus –giving support to producer groups in these)

Outputs of MAGIP

• In 11 remote gewogs where communities have no or difficult road access 880 households supported with low cost labor saving technologies in subsistence farming

• In 37 gewogs with good road access and agro-ecological potential project to support 3600 producers organized in groups with training inputs and services for cash crops and dairy production

• Marketing and storage infrastructure, farm roads and irrigation schemes constructed or rehabilitated

Broad characteristics of households in MAGIP area

AOS survey –closest collaboration between Bhutan project team/IFAD and IFPRI• Work collaboratively to develop questionnaire

• Try to make it more quantitative

• In fact try to get a mix of quantitative and qualitative

• Work collaboratively for sampling• Could have longitudinal data

• Data collection methods • Digitalization of survey

• Possible link with questionnaire development

Results based Monitoring and Evaluation under MAGIP

• All RGOB funded programs routine M&E activities are responsibility of district staff

• Gewog level RNR extension staff should regularly collect data at activity, output, outcome and impact levels

• Results data to be entered into the web based PLAMS

• RIMS first and second level indicators submitted annually to IFAD

• Given the difficult geographical terrain standard RIMS impact surveys not to be conducted, alternative methodology to be devised

AOS questionnaire in Bhutan

• FinalforShanti_docx

• Get household identification- some more details –why

• Try tracking households- why?

• Mix of quantitative and qualitative questions why?