Randomized Evaluation: Start to Finish - The Abdul Latif .... Start to... · Evaluation from Start...

Randomized Evaluation:

Start to Finish

Freida Siregar

J-PAL SEA

Course Overview

1. What is evaluation? Why Evaluate?

2. Why randomize?

3. How to randomize

4. Threats and Common Pitfalls

5. Evaluation from Start to Finish

6. Cost Effectiveness Analysis

Course Overview

1. What is evaluation? Why Evaluate?

2. Why randomize?

3. How to randomize

4. Threats and Common Pitfalls

5. Evaluation from Start to Finish

6. Cost Effectiveness Analysis

Evaluation Overview

4

Evaluation Design

• Context

• Research/ Policy question

• Treatments

• Randomization

Baseline

• Survey: Collect indicators

• Random assignment

• Quality checks

Treatment

• Treatment implementa-tion

• Internal controls

• Coordination

• Field visit

Midline and/or Endline

• Survey: Collect indicators

• Quality checks

Analysis

• Policy findings

• Policy outreach

Evaluation from Start to Finish

5

1. Background and Design

2. Implementation: Treatment

3. Implementation: Data Collection

4. Analysis and Scale up

Evaluation background

6

Background and Design

7

Context

• Raskin program

• Partner: TNP2K

Research/ Policy question

• Research/ Policy question

• Theory of change

• Log frame

Treatments Randomization

• Unit of randomization

• Sample frame

• Stratification


8

Context

• Raskin program

• Partner: TNP2K


• Research/Policy question


• Log frame



• Sample frame

• Stratification

The program: Raskin

9

• One of Indonesia’s largest social assistance programs – US $1.5 billion spending each year

– 53% of national public expenditure on social assistance (World Bank 2012)

• Provides poor and vulnerable households with subsidized rice – 15 kg of rice sold for Rp. 1,600 per kg at Distribution Point

(Titik Distribusi)

– Targeted to Indonesia’s poorest 30% (PPLS’10)

• In practice, eligible households only received 32% of the intended subsidy (Raskin HH survey)

The partner: TNP2K

10

• Tim Nasional Percepatan Penanggulangan Kemiskinan (TNP2K),

• Established by and positioned under Vice President Boediono

• TNP2K’s mandate: – Broker evidence-based policies for

improving effectiveness of social assistance programs,

– Coordinate multiple government actor towards this purpose

• Improved program targeting and delivery is a major priority for TNP2K. Raskin falls under TNP2K’s Cluster I


11

Context

• Raskin program

• Partner: TNP2K




• Log frame



• Sample frame

• Stratification

Evaluation Objectives

12

Build evidence about whether TNP2K’s proposed

distribution of Raskin ID cards will improve the Raskin

program

The pilot aims to answer three key policy questions:

1. Will Raskin cards improve Raskin targeting of and

take-up rates for eligible households (RTS-PM), and

reduce price?

2. Will Raskin cards be socially accepted?

3. What is the most effective way to implement cards?

Theory of Change

Distribution of Raskin ID cards to Raskin

beneficiaries

Reception of Raskin ID cards

Beneficiaries obtain more of

their Raskin subsidies

Increase effectiveness of social protection

programs

Sample assumption: Beneficiaries understand use of cards, use cards, do not confuse with previously existing Raskin cards/coupons

Sample assumption: Beneficiaries demand reduced prices, village Raskin authorities listen to beneficiary demands and have ability to make change

Sample assumption: Successful delivery to household, no “bypassing” of cards

Sample assumption: Fair(er) practices sustained, poor program effectiveness due to lack of transparency

Log Frame

14

Objectives Hierarchy Indicators Sources of Verification Assumptions / Threats

Impact (Goal/ Overall

objective)

Increase

transparency and

effectiveness of

social protection

programs

Quantity and price of Raskin purchased

Household survey Fair(er) practices sustained, poor program effectiveness due to lack of transparency

Outcome (Project

Objective)

Beneficiaries

obtain more of their

Raskin subsidies

Quantity and price of Raskin purchased

Household survey Beneficiaries demand reduced prices, village Raskin authorities listen to beneficiary demands and have ability to make change

Outputs Reception of Raskin ID cards

Whether or not beneficiaries receive Raskin cards

Household survey Beneficiaries understand use of cards, use cards, do not confuse with previously existing Raskin cards/coupons

Inputs (Activities)

Distribution of Raskin ID cards to Raskin beneficiaries

Whether or not cards are sent

Household survey, administrative data from PT Pos*

Successful delivery to household, no “bypassing” of cards


15

Context

• Raskin program

• Partner: TNP2K




• Log frame



• Sample frame

• Stratification

Intervention Overview

16

Raskin ID Card

Variation 1: Design

Variation 2: Content

Variation 3: Distribution

Variation 4: Socialization

Coupons

No coupons

Price at TD

No Price

All beneficiaries

Bottom 10%

Standard

Enhanced

Example of Raskin ID Cards

17

Raskin card with coupon and price Raskin card without coupon, with price

Enhanced Socialization Posters

18

Treatment variations

19

Card Variations Standard

socialization Enhanced

Socialization

All beneficiary

Price Coupon Group 1 Group 2

No Coupon Group 3 Group 4

No Price Coupon Group 5 Group 6


Bottom 10%





Control (No card, no socialization)

Treatment Implementation Sept – Nov ‘12

Data Collection

Midline Survey Oct – Dec ‘12

Endline Survey Mar – May ‘13

Project Implementation

Baseline Jan – Feb ‘12

Dec 2012 presentation

June 2013 presentation

Process & Impact Evaluation

20

Design Evaluation


21

Context

• Raskin program

• Partner: TNP2K




• Log frame



• Sample frame

• Stratification

Identifying unit of randomization

22

• What is the smallest

administrative unit at which

Raskin distribution is

conducted?

• Kecamatan? Gudang bulog?

Village? Dusun?

Source: www.bulog.co.id

Province

City/Regency

Village

Sub-city /regency

Sub-city /regency

Governor

Mayor/Regent

Perum Bulog (Divre/Subdivre/Kansilog)

Gudang (Satgas Raskin)

Titik Distribusi (Pelaksana Distribusi)

Pokja Warung Desa Pokmas

Raskin HH beneficiaries (Paid in cash, Rp 1.600/kg netto at the

Distribution Point)

Head of National Raskin Team (Kemeko Bid Kesra)

Sample frame

23

• 600 villages (including control)

28 dropped due to risk and remoteness

• 572 villages within 6 Kabupaten

Pemalang and Wonogiri (Central Java),

Palembang and Ogan Komering Ilir (South Sumatera),

Bandar Lampung and Central Lampung (Lampung)

• Raskin sample

identical to

previous project’s

(Targeting II)

Stratification

24

• Treatment stratified by

– Kabupaten,

– Targeting II Treatment Group,

– Kecamatan and

– Urban to rural ratio of 2:3

What the randomization looks like

25

Treatment 1: Raskin ID Card

Treatment 2: Raskin ID Card + Enhanced Socialization

Control: No treatment

26

Treatment and Control villages were statistically

equivalent prior to the pilot

Baseline data from February 2012

Control CardsN = 5643

03

69

By Treatment Status

Raskin purchased (in kg)

IMPLEMENTATION:

TREATMENT

27

Implementation: treatment

28

ID how to implement

treatment on ground

Establish strong internal controls

Coordination with

government

Prep, launch treatment

(cards, enh soc) Visit field

Implementation Plan

29

• Identify how to implement treatment on ground

– Develop detailed implementation plan with facilitation

specialist and government

– Consider scale up: utilize government’s vendors of choice

• Establish strong internal controls

– Random checks on card printing

– Recruitment standards (for facilitators)

– Standardized forms to document treatment implementation

– Clear reporting procedure

Implementation Plan

30

• Coordination with government – Jakarta workshop for reps from province, kabupaten/kota

– Pre-field work coordination with regency/city, district, and village heads

• Prepare, launch treatment

– Penjadwalan, pelatihan, koordinasi logistik

• Visit field: qualitatively observe the responses to treatment

Treatment variations

31

Card Variations Standard

socialization Enhanced

Socialization

All beneficiary





Bottom 10%





Control (No card, no socialization)

32

Recap: Raskin ID Card Variations

Cards randomly varied by:

• Design: coupon or no coupon

• Content: price or no price

• Distribution: to all Raskin beneficiaries or only the

poorest 10%

Distribute to Beneficiaries

• Sept. to mid-Oct. ‘12

• 378 villages receive cards;

194 villages control

33

Cards villages randomly varied to receive:

• Standard Socialization: letter, DPM; or

• Enhanced Socialization:

• + 3 DPM per dusun

• + 3 information posters per dusun

• Socialization to village leaders

• Announcement through speakers


34

• Late-Sept. to mid-Nov. ’12

• The 378 villages that received cards

received socialization

• 186 villages: Standard Socialization

• 192 villages: Enhanced Socialization

• The 194 control villages did not receive

socialization


A facilitator explaining Raskin Cards to village leaders in OKI, Central Lampung

Challenges in treatment implementation

35

• Some treatment villages were

unsafe or very remote

• Many cards did not reach

beneficiaries

Challenges in treatment implementation

36

• Facilitators did not receive enough Raskin

beneficiary posters

• During meetings, participants complained about other aspects of Raskin/ government programs

IMPLEMENTATION: DATA

COLLECTION

37

Implementation: data collection

38

Draft questionnaire

Internal Piloting

Coordination with

government

Prepare, launch data collection

teams

Quality checks, Field visits

Data Collection Plan

39

• Survey instruments: household and community survey

• Baseline—use previous project’s endline – Ensure control and treatment groups are statistically equivalent

• Identify respondents – Respondents:

• Raskin beneficiaries (poor )

• Raskin beneficiaries (very poor; 10% paling miskin)

• Non-Raskin beneficiaries

– Listing, to identify non-Raskin beneficiary households

– Use PPLS’10 data, to identify potential Raskin beneficiary households

Challenges in data collection

40

• Matching administrative data with field

– Human error, change in poverty status/address

• Accommodating changes to administrative areas (e.g.

pemekaran)

• Time constraints

• Obtaining sufficient HR to conduct data collection work

• Other concerns: how far behind can respondents

remember? How do we phrase/explain “Titik Distribusi”?

Etc.

Data Collection

41

Survey Data source Respondents surveyed

Data collected

Baseline 2011

Endline from Targeting II, a previous project

PKH eligibles, non-poor

Main purpose: ensure control and treatment groups statistically equivalent

Midline Oct-Dec’12

5,148 HH, through household surveys and community surveys (targeted to village head)

Mix between non-poor and Raskin beneficiaries (poor and very poor)

Amount and price of Raskin rice purchased, awareness on Raskin program, satisfaction levels with Raskin program, overall household consumption, subjective wealth/standing, etc.

Endline Mar-May ’13

6,292 HH, through HH and community surveys

Ibid Ibid

Challenges in program evaluation

42

• Attrition: when evaluators fail to collect data on

individuals who were selected as part of the original

sample

– ML: 9% replaced (418/4,572), EL: 9.8% (561/5,706)

– Replacement of respondents integrated into data collection

process

– Dropped 28 villages

43

ANALYSIS

44

Analysis

45

1. Write analysis plan 2. Write STATA do.file 3. Run data through program

4. Generate graphs 5. Analyze: Connect and reference with qualitative field observations

6. Disseminate findings

Higher satisfaction with Raskin by

eligible households in card villages

relative to the control

Increase in purchase of Raskin by

eligible compared to control

Eligible households in treatment villages

purchase more rice than those in control group

46


02

04

06

08

0

Eligible


Ineligible

By Treatment Status and Sample

HH bought raskin (pooled since oct)


01

.93.8

5.7

7.6

Eligible


Ineligible

By Treatment Status and Sample

Amount of raskin (kg, pooled since oct)

Can we generate this “subsidy received” graph?

1.4kg (25%) more rice

4.5pp

MID

LIN

E R

ESU

LTS

47


receive more in subsidy

No difference in household satisfaction

between treatment and control


purchase more rice


pay a smaller markup price

0.9 kg (19%) more rice

Rp. 77/kg (13%) smaller markup price

approx. Rp. 6000/HH/month (21%)

So did the ineligibles in the card villages also enjoy a smaller markup price? How much?

END

LIN

E R

ESU

LTS

Evolution of Results

48

Treatment → Outcome Midline Endline Cards → take-up, purchase, and subsidy

(+) (+)

Cards → HH satisfaction (+) eligible no effect

Enhanced Socialization (ES)

→ purchases, subsidy (+) eligible (+) eligible

ES → satisfaction (+) HHs (+) HHs

(−) leaders no effect leaders

Printing price → price markup (−) (−), only in ES

Coupons → subsidy (−) ineligible (+) eligible

Distribution to poorest 10%

→ rice purchases no effect (+), due to ES

49

• Distributing Raskin Cards improves the program

• Beneficiary take-up and Raskin purchases increase

• Price markup decreases

• On net, subsidy of Rp. 6000 for eligible with no

decrease for ineligibles

• Enhanced socialization improved beneficiary subsidy

and satisfaction

• Achieved with only 2-3 person-days of external

facilitation and 3 posters per dusun.

Insights for Policy

50

• Printing price on ID cards improved their effectiveness

• Short-run effect persists in the medium-run, when

combined with enhanced socialization

• Distributing cards only to the poorest 10% may be an

effective way to improve targeting

• Initially, poorest decile reported lower satisfaction and no

difference in take-up

• However, in the latest survey they report a higher subsidy

and higher satisfaction, in villages with enhanced

socialization

Insights for Policy (cont.)

51

• Coupon usage potentially effective with enhanced

socialization

• In the medium run, card variation effects depend on

enhanced socialization of the program.

• This applies to printing the price, coupons and

targeting the poorest 10%

Insights for Policy (cont.)

Policy Scale Up

52

Social Protection Cards (KPS)

53

• TNP2K scaled up Raskin Cards as Social Protection Card (KPS).

• As of June 2013, KPS cards have been distributed nationally to 15.5

million households (65.6 million people).

• The cards can be used access a temporary unconditional cash transfer

(BLSM); a cash transfer for poor students (BSM); and Raskin.

TERIMA KASIH

54

APENDIX

55

The Research Team

56

Principal

Investigators Design treatment,

randomization, analysis

Research Analyst Randomization, generate

respondent list, STATA

coding

Research Associate Develops implementation,

manages project (budget,

timeline, relationships,

reports), quality checks

Data Manager De-identifies data, generate

respondent list, cleans data,

matching, data quality

checks

Date post:	01-Feb-2018
Category:	Documents
Upload:	trinhnhan
View:	224 times
Download:	0 times

Randomized Evaluation: Start to Finish - The Abdul Latif .... Start to... · Evaluation from Start...

Documents