Investing in Indonesia’s Institutions
for Inclusive and Sustainable Development
PKH Evidence & Policy
Implications: summary of
Results from Impact
Evaluation, Operations
Analysis, and Spot Checks
Diskusi Pokja Kebijakan Monev TNP2K
29 March 2011
1Kebijakan Monev 1
Intro Methods Poverty Health Educ Labor Consumption Policy
Summary of Results from PKH Impact Evaluation
− Methodology
− Poverty
− Health
− Education
− Child Labor
− Consumption
Further evidence from Spot Checks and SMERU qualitative study
Policy going forward: discussion
− Education
− Supply readiness
− Facilitators
− Affiliated providers
Today’s topics
Intro
2Kebijakan Monev 2
Intro Methods Poverty Health Educ Labor Consumption Policy
PKH was randomized to kecamatan, non-random to households.
Methods
Randomization is effective at ensuring that treatment decisions and response to treatment are orthogonal and uncorrelated. “
”
to which should be added “including unobservable characteristics and unmeasured trends”
With equivalent distributions of characteristics we certain that treated and untreated groups are valid counterfactuals for each other:
“Equivalent distributions of characteristics” are tested with baseline (pre-program) data on observable characteristics.
3Kebijakan Monev 3
Intro Methods Poverty Health Educ Labor Consumption Policy
PKH was randomized to kecamatan, non-random to households.
The PKH kecamatan and household-selection processes:
Methods
4Kebijakan Monev 4
Intro Methods Poverty Health Educ Labor Consumption Policy
PKH was randomized to kecamatan, but not 100% according to plan
Beware #1: the PKH kecamatan selection process was contaminated:
− 39 out of 180 (22%) surveyed lottery control kecamatan were found with PKH
− 2 out of 180 (1%) surveyed lottery treatment kecamatan were found to never have received PKH
Methods
5Kebijakan Monev 5
Intro Methods Poverty Health Educ Labor Consumption Policy
PKH was non-random to households in an observable way
Beware #2: the household selection processes was purposive:
Methods
6Kebijakan Monev 6
Intro Methods Poverty Health Educ Labor Consumption Policy
Beware #2: the household selection processes was purposive:
Result #1: PKH households worse off compared to even other eligible households
− Younger, larger, female-headed, in agriculture
− Less education, fewer assets, more likely to be BLT and Jamkesmas beneficiaries (but scholarships don’t matter)
− Less connected to village authorities and less involved in governmental groups
PKH was non-random to households in an observable way
Methods
7Kebijakan Monev 7
Intro Methods Poverty Health Educ Labor Consumption Policy
To continue using full matched-at-baseline sample while incorporating lottery/actual mismatches:
− Use lottery assignment as instrument for actual status
− Control (probabalistically) for those who received and missed incorrectly
To compare treated households to like households in control kecamatan
− Use indices reflecting likelihood of receiving PKH; build indices from observed characteristics
− Control (probabalisitically) for those who would have received had PKH been in their kec
To provide an alternative specification (robustness)
− Difference-in-differences method with two alternatives (xsection or matched panel)
− Provide estimates of behaviors over randomly-receiving or randomly not-receiving hh only
Fixes for kecamatan contamination and non-random household selections
What difference in
differences (DID) does:
subtract from pre- to
post-PKH changes the
same over-time change
in non-PKH areas.
Methods
8Kebijakan Monev 8
Intro Methods Poverty Health Educ Labor Consumption Policy
Average baseline (2007) expenditure (household average, per capita per month): IDR 199,000− average for all eligible households; eventual PKH households slightly poorer at baseline
Average PKH household increase in expenditure (2007 to 2009): IDR 19,000 (9.5% of baseline)
− this is over and above what similar non-PKH families in control kecamatan experienced.
PKH was enough to move ~700,000 eligible beneficiaries out of poverty (on average)
− March 2009 poverty line: IDR 200,262
PKH share in overall measured poverty reduction could have been 82% (on average) :− 2007→2009: headcount poverty rate 16.6% → 14.1% implies 858,000 fewer poor hh
Average baseline expenditure (2007) for eventual PKH households : IDR 184,000− Average PKH impact on these households (IDR 19,000) = 10% of baseline
PKH moved ~700,000 actual beneficiaries to the March 2009 poverty line− implicit reduction in poverty gap/severity: 1.9/0.74 index points
PKH share in decrease in poverty gap/severity could have been 73%/65% (on average):
− 2007→2009: actual poverty gap 5.1 → 2.5 and actual severity 1.8 → 0.675
PKH generated significant reductions in poverty, poverty gap measures
Poverty
1 Implied reductions in poverty measures
Note: all results approximately the same under the assumption that the full value of an average 2008 PKH transfer (IDR 1.2 million) is used (by a PKH household with an average of 5.2 members) on expenditure items used to calculate total household expenditure. Such an assumption implies an average increase in expenditure of IDR 19,231 per-capita per month.
9Kebijakan Monev 9
Intro Methods Poverty Health Educ Labor Consumption Policy
Many Healthy Behaviours saw significant and positive increases
Health
2a Significant Health Behaviour Changes
10Kebijakan Monev 10
Intro Methods Poverty Health Educ Labor Consumption PolicyHealth
Many Healthy Behaviours saw significant and positive increases
2a Significant Health Behaviour Changes
11Kebijakan Monev 11
Intro Methods Poverty Health Educ Labor Consumption PolicyHealth
Many Healthy Behaviours saw significant and positive increases
2a Significant Health Behaviour Changes
12Kebijakan Monev 12
Intro Methods Poverty Health Educ Labor Consumption PolicyHealth
2a Significant Health Behaviour Changes – Spillovers to non-PKH hhs
Many Healthy Behaviours saw significant and positive increases
13Kebijakan Monev 13
Intro Methods Poverty Health Educ Labor Consumption PolicyHealth
2a Significant Health Behaviour Changes
Many Healthy Behaviours saw significant and positive increases
14Kebijakan Monev 14
Intro Methods Poverty Health Educ Labor Consumption PolicyHealth
2a Significant Health Behaviour Changes – further evidence
Healthy Behaviours benefitted from significant promotion
SMERU Qual Study - Java Barat and NTT; 24 desa/kelurahan); Java Barat accessible; NTT remote
For maternal & child care (MCH), PKH led to ↑
− Posyandu attendance (primarily NTT)
− attendance at Posyandu alternatives (primarily urban)
− awareness of and activity on MCH issues both among beneficiaries and village officials, midwives, posyandu cadres, community leaders, religious leaders, NGOs
− Facilitators crucial: positive encouragement and negative threats
PKH did not coincide with or lead to
− ↑ access to MCH for remote villagers
Factors mentioned: no incentives and/or logistical support for posyandu cadres; absent midwives; poor roads and rainy-season shutdown; lack of public transport
− ↑ community involvement in MCH budget or provision decisions
15Kebijakan Monev 15
Intro Methods Poverty Health Educ Labor Consumption Policy
Not all Healthy Behaviours saw significant increases for PKH hhs
Health
2b Not Significant Health Behaviour Changes
16Kebijakan Monev 16
Intro Methods Poverty Health Educ Labor Consumption PolicyHealth
2b Not Significant Health Behaviour Changes – further evidence
Provider quality and remoteness contributed to remaining gaps in health
CHRUI Spot Checks - Java (East, West, Jakarta), South Kalimantan, NTB; 36 kecamatan; 1800 mothers, 108 facilitators, 108 schools, 36 Puskemas,
PKH did not coincide with or lead to
− ↑ access to puskemas services for remote villagers
Factors mentioned: distance, bad transport, no transport available
− ↑ access to comprehensive service at posyandu (materials and tools; information and copies of records; complete schedule of treatments/interventions)
− ↓fees at puskesmas
− ↑ grievance redress when services rejected or delayed by health service providers.
− full availability of supplies at health service providers
Human capital: dentists, midwives, public health officer, paramedics, nutrionist, pharmacist not always available at puskemsas
Vaccines& Vaccine storage: occasionally missing at puskemsas
17Kebijakan Monev 17
Intro Methods Poverty Health Educ Labor Consumption PolicyHealth
2b Not Significant Health Behaviour Changes – evidence in urban or densely populated Java
Provider quality and remoteness contributed to remaining gaps in health
18Kebijakan Monev 18
Intro Methods Poverty Health Educ Labor Consumption Policy
Medium- to long-term health outcomes are mostly showing no changes yet
Health
2c Mostly Not Significant Health Outcomes
19Kebijakan Monev 19
Intro Methods Poverty Health Educ Labor Consumption Policy
But health-seeking behavior has already changed…
Health
2c Mostly Not Significant Health Outcomes – evidence from urban, dense, and agricultural areas
20Kebijakan Monev 20
Intro Methods Poverty Health Educ Labor Consumption Policy
High baseline levels in Education make further increases difficult
Educ
3 Mostly Not Significant Education Outcomes
21Kebijakan Monev 21
Intro Methods Poverty Health Educ Labor Consumption Policy
Low levels of Wage work make further reductions difficult
4 Mostly Not Significant Child Labor Outcomes
Labor
22Kebijakan Monev 22
Intro Methods Poverty Health Educ Labor Consumption Policy
3 Insignificant Education Changes – further evidence
Education supply remained inadequate; advocacy was insufficient
For primary education
− Enrollment did not increase because of previously high levels of attendance
− Facilitators crucial for ↑ in Hours/Attendance: positive encouragement and negative threats
note: SMERU finds more effective facilitators in NTT b/c of reduced range, not reduced caseload
For secondary education
− Enrollment increased only in NTT (again, facilitators crucial)
− Facilitators crucial for ↑ in Hours/Attendance
PKH did not coincide with or lead to
− ↑ access to 2ndary schools
Factors mentioned: remoteness/distance to schools; lack of public trans; overall cost; early marraige/pregnancy; job demands (harvest)
− ↑ supply of quality at 2ndary schools
Factors mentioned: overcapacity (from BOS); decline in parents’ participation (again from BOS); low levels of teacher education; lack of facilities (libraries, labs, teaching aids, electricity, clean water)
SMERU Qualitative
In education, PKH led to ↑
− Awareness among beneficiaries and village officials, midwives, posyandu cadres, community leaders, religious leaders, NGOs
− Awareness of links from education to employers’ requirements and better future
Educ
23Kebijakan Monev 23
Intro Methods Poverty Health Educ Labor Consumption Policy
3 Insignificant Education Changes – further evidence
Education supply remained inadequate; advocacy was insufficient
CHRUI Spot Checks
PKH did not coincide with or lead to
− ↑ participation or attendance
Factors mentioned: attendance rates already approx. 100%.; no outreach to those who leave school early; PKH did not cover opportunity costs of secondary school
− ↑ access to 2ndary schools
Factors mentioned: remoteness/distance to schools – (2km/22min on average, 20km/120min maximum to reach with foot or pedal power as the most common mode of transport); no tuition waivers;
− ↑ supply of quality at 2ndary schools
Factors mentioned: low levels of teacher education; overcrowding; lack of facilities (libraries, labs, teaching aids, electricity, clean water)
Educ
24Kebijakan Monev 24
Intro Methods Poverty Health Educ Labor Consumption Policy
Region & Sector trends are encouraging; household trends less so
3,4 mostly not significant Education & Child Labor – further evidence
LaborEduc
25Kebijakan Monev 25
Intro Methods Poverty Health Educ Labor Consumption Policy
Consumption, especially of health goods and services, increases
5 Significant Consumption Increases
Consumption
26Kebijakan Monev 26
Intro Methods Poverty Health Educ Labor Consumption Policy
Consumption, especially of health goods and services, increases
5 Consumption demonstrates significant increases in health seeking behaviour
note: does absence of change in
education spending indicate
anything re: hh constraints to
education?
Consumption
27Kebijakan Monev 27
Intro Methods Poverty Health Educ Labor Consumption PolicyPolicy
Education – are PKH benefits enough to expect improvements given costs?
PKH benefits from attention; remaining reforms become clear
SMP/SMA costs per year for poor hh: ~IDR 2.5 million or ~30% of expenditure; PKH households generally did not receive scholarships
PKH family size:5.2 members, 3-4 children needing care
28Kebijakan Monev 28
Intro Methods Poverty Health Educ Labor Consumption Policy
Education – or is it payment timing and program design?
PKH benefits from attention; remaining reforms become clear
2030
4050
6070
8090
100
Per
cent
(%
)
0 1 2 3 4 5 6 7 8 9 10
Years of Education
quintile 1 quintile 2 quintile 3 quintile 4 quintile 5
Education Attained by IndonesianAge 26-28, Year 2009
Graduated SD/MI
GraduatedSMP/MTs
In the beginning, the 4-per year payment design was changed to 3-per year. As a consequence, there was noPKH payment synchronized with due dates for school fees. These modifications were partially due to the inability of UP-PKH and PT Pos to manage keep the
validation and verification process timely (throughput of forms and data)
Policy
29Kebijakan Monev 29
Intro Methods Poverty Health Educ Labor Consumption Policy
Education – or is it in implementation?
PKH benefits from attention; remaining reforms become clear
Socialization to affiliated providers was weak
− Providers may not have understood that conditionalities (and compliance monitoring) wasn’t important and the PKH was not just “business as usual” for them
See process engineering assessment and CHRUI for factors behind weak socialization activities
Facilitators were not as effective at encouraging education
− Facilitators and the community generally did not make effort to re-insert drop-outs into the system; facilitators were sometimes too diffuse to be able to visit remote households as soon as drop-outs were noted
See CHRUI for factors behind inability of facilitators to improve education behaviors where they were too disperse
Affiliated agencies did not isolate PKH households for monitoring or assistance
− PKH households generally did not receive scholarships or subsidized transportation
Policy
In reality, all of these issues are at work and all need serious thought
Weakness in one area (form processing) has knock-on effects in other areas (altered payment schedule) which can affect outcomes (inability to recapture drop-outs or improve transition rates)
30Kebijakan Monev 30
Intro Methods Poverty Health Educ Labor Consumption Policy
Relative to other Indonesia SSN, PKH expenditures appear appropriate
1 PKH expenditures show appropriate shares for benefits relative to support operations
WB Social Assistance Public Expenditure Review
Policy
31Kebijakan Monev 31
Intro Methods Poverty Health Educ Labor Consumption Policy
1 PKH expenditures show appropriate shares for benefits relative to support operations
WB Social Assistance Public Expenditure Review
PKH’s operation costs have come down, similar to international examples
Other facilitated programs spending similar amounts on operations
Other programs with similar coverage or similar benefits (Jamkesmas, Scholarships) spend very little on non-benefit operations
− These programs are not facilitated
− These programs do not reach only poor households
− These programs have larger impacts for non-poor households.
− These programs rely on local-level administration (at the facility) to run the program – they are essentially transfers from central government agencies to service providers.
Relative to other Indonesia SSN, PKH expenditures appear appropriate
Policy
32Kebijakan Monev 32
Intro Methods Poverty Health Educ Labor Consumption Policy
Qualitative Studies raised several additional operational issues
2 Validation and updates to household rosters
CHRUI Spot Checks
− 18% households could not be re-verified; 69% had Jamkesmas; only 18% had Scholarships; mismatches between UP-PKH pusat databases and actual recipients;
− 84% households reported demographic changes; updates were not monthly everywhere
Factors mentioned: lack of personnel; lack of forms; power outages; non-functioning MIS (including automatic checks by province or central level); inaccurate BPS data
− Solutions
uji coba (pilot phase): Targeting improvements and recertification strategies; MIS fully functional (and staffed?) everywhere; Form printing and delivery supply chain improvements
Building in data sweeps for mismatches to occur some weeks before payment process begins
Policy
33Kebijakan Monev 33
Intro Methods Poverty Health Educ Labor Consumption Policy
3 Compliance verification and conditioning
CHRUI Spot Checks
− 71%, 68%, 29%, 24%, 19% with BCG, measles, polio, DPT, and hepatitis vaccines (respectively); 2.2% not going to school; and no recorded changes in benefits.
Factors mentioned: mismatch between names of mothers/children reported at local level and in UP-PKH pusat database; forms content, printing, and delivery problematic; lack of incentives (mainstreaming); lack of socialization/enforcement on conditionalities (including to affiliated operators like PT Pos)
− Solutions
sudah: MIS fully functional nearly everywhere; CVS system operational for 85% of hh (@ Jan 2010, increasing lately)
uji coba (pilot phase): Form printing and delivery supply chain improvements (joint with PT Pos and facilitators)
Resocialization and re-examining socialization schedule; audit of socialization process.
sharing lessons with affiliated providers (MoNE, MoRA, MoH); incentive schemes or enforceable MOUs with affiliated providers?
Qualitative Studies raised several additional operational issues
Policy
34Kebijakan Monev 34
Intro Methods Poverty Health Educ Labor Consumption Policy
4 Payment timeliness, scheduling, and payment processing schedules
CHRUI Spot Checks
− 75% report correct payment amounts and 95% received exactly the receipt amount; some districts scheduled 2 or 3 tranches per year instead of 4.
Factors mentioned: CVS and data updates not operating; data mismatch.
− Solutions
sudah : “4 per year” schedule enforced; timing tranches to school fees schedule; MIS fully functional nearly everywhere
uji coba (pilot phase): Form printing and delivery supply chain improvements; CVS system operational for 85% of hh (@ Jan 2010)
Qualitative Studies raised several additional operational issues
Policy
35Kebijakan Monev 35
Intro Methods Poverty Health Educ Labor Consumption Policy
Process Engineering examines in detail the “PKH machine”
5 Socialization & Complaints and Grievances systems
Ayala Consulting (& confirmed in CHRUI Spot checks)
− Socialization delegated to KemenKomInfo not adequately carried out (to beneficiaries and affiliated providers) either in content, frequency, or intensity; no printed materials; officers not able to adequately answer questions; lack of monitoring of socialization process resulted in lingering misunderstanding; PKH beneficiaries not thought elibible for Raskin or Jamkesmas; most info via word of mouth or media reports of malfeasance
− Majority of beneficiaries did not know how to file complaint and were unaware of the CGS; written complaints too costly (fear of losing benefits or faux pas); lack of forms; no response on the small amount of complaints received; MIS system does not automatically respond to incoming grievances or complaints
− Potential Solutions
uji coba (pilot phase): recertification and “waiting lists” in the targeting exercise
Policy
36Kebijakan Monev 36
Intro Methods Poverty Health Educ Labor Consumption Policy
Process Engineering examines in detail the “PKH machine”
5 Socialization & Complaints and Grievances systems
− Potential Solutions (continued)
DepSos, Bappenas, or TNP2K recapture socialization mandate in order to deliver better socialization, policy, materials & effort
Case Management system: facilitators as “one-stop shop” for all social assistance initiatives for eligible households (April 17-29 – re-visit by int’l consultant)
Affiliated agency MOUs: positive and negative incentives to generate interest and effectiveness in affiliated service providers (PT Pos, KemenKomInfo, MoNE, MoRA, MoH) (April 17-29 – re-visit by int’l consultant)
Review of current CGS procedures in manual and investigation of non-deployment (April 17-29 – re-visit by int’l consultant)
Policy
37Kebijakan Monev 37
Intro Methods Poverty Health Educ Labor Consumption Policy
6 UP-PKH organization and cross-agency organization is ineffective
Ayala Consulting & int’l experts (& confirmed in CHRUI Spot checks)
− Affiliated agencies and local gov’ts agencies consider PKH a burden and do not instruct regional offices to give continuous support; coordination meetings not attended; DepSos lacks authority and/or appropriate agreements to monitor progress (and remedy lack of progress) in affiliated agencies (in 2010, no PKH-related activities were undertaken by KemenKomInfo until earliest August); MOUs inappropriate and unenforceable
− regional coordinators stuck in Jakarta; DepSos lacks capacity/authority to implement a full-scale continuous reform cycle (improvements at the margin); no performance evaluations carried out by UP-PKH; no TA or monitoring available at/from UP-PKH propinsi level (this is inadequate for a country as large and diverse as Indonesia); unclear responsibilities for regional and provincial coordinators; PKH has no specialized personnel for policy/planning, training, inter-agency coordination, M&E and continuous reform.
− Potential Solutions
sudah : Korwil moved back to province
Process Engineering examines in detail the “PKH machine”
Policy
38Kebijakan Monev 38
Intro Methods Poverty Health Educ Labor Consumption Policy
6 UP-PKH organization and cross-agency organization is ineffective
− Potential Solutions (continued)
Affiliated agency MOUs: positive and negative incentives to generate interest and effectiveness in affiliated service providers (PT Pos, KemenKomInfo, MoNE, MoRA, MoH) (April 17-29 – re-visit by int’l consultant)
Review of current UP-PKH organization and feasible blueprint improvements (April 17-29 – re-visit by int’l consultant)
TNP2K to raise PKH’s profile; point out interest other agencies may have; point out the wealth of data indicating weaknesses in cross-agency collaboration
Process Engineering examines in detail the “PKH machine”
Policy
39Kebijakan Monev 39
Intro Methods Poverty Health Educ Labor Consumption Policy
Issue #6: Organization within UP-PKH
Placeholder for Ayala’s current
Process Engineering examines in detail the “PKH machine”
Policy