Page 1 of 25 PNPM SUPPORT FACILITY (PSF) Project Proposal Project Title: PNPM Monitoring and Evaluations (M&E) and Special Studies (2013-2015) Development Objectives: The objectives of the PNPM M&E trust fund program are to: (i) provide stakeholders with empirical data regarding results and impact of PNPM Mandiri; (ii) conduct research in greater depth regarding special topics of concern to PNPM and the field of social development; and, (iii) support capacity building efforts for Indonesian research organizations. Key Performance Indicators Data and findings used to improve implementation and enhance design/impact in existing and new programs Evidence and recommendations regarding sustaining poverty- reduction and pro-poor growth impacts of CDD programs delivered to PNPM stakeholders Evidence and recommendations regarding PNPM, local economy and livelihoods delivered to stakeholders Mapping of community groups’ contributions to accountability in achievement of minimum standards service delivery, delivered to stakeholders Evidence regarding local constraints to adopting PNPM governance practices delivered to practitioners’ forums Recommendations regarding strategies for increasing inclusion and outreach to marginalized households and groups delivered Research activities, products, and services planned for and developed in collaboration with PNPM/Cluster 2 policy-setting stakeholders Number of studies completed as part of the 2013-2015 round of evaluations Analytic pipeline proposal for future work to address new frontiers/special topics of concern to PSF stakeholders Capacity-building action plan for Indonesian research partners developed and implemented Short-course training modules – focused on technical details in implementing qualitative and/or quantitative social science research – delivered Executing Agency: PSF Secretariat/World Bank Estimated Budget: US$ 4,750,000 Estimated Duration: 32 months (covering activities to December 31, 2015)
Transcript
Page 1 of 25
PNPM SUPPORT FACILITY (PSF)
Project Proposal
Project Title:
PNPM Monitoring and Evaluations (M&E) and Special Studies
(2013-2015)
Development
Objectives:
The objectives of the PNPM M&E trust fund program are to:
(i) provide stakeholders with empirical data regarding results and
impact of PNPM Mandiri;
(ii) conduct research in greater depth regarding special topics of
concern to PNPM and the field of social development; and,
(iii) support capacity building efforts for Indonesian research
organizations.
Key Performance
Indicators Data and findings used to improve implementation and enhance
design/impact in existing and new programs
Evidence and recommendations regarding sustaining poverty-
reduction and pro-poor growth impacts of CDD programs delivered
to PNPM stakeholders
Evidence and recommendations regarding PNPM, local economy and
livelihoods delivered to stakeholders
Mapping of community groups’ contributions to accountability in
achievement of minimum standards service delivery, delivered to
stakeholders
Evidence regarding local constraints to adopting PNPM governance
practices delivered to practitioners’ forums
Recommendations regarding strategies for increasing inclusion and
outreach to marginalized households and groups delivered
Research activities, products, and services planned for and developed
in collaboration with PNPM/Cluster 2 policy-setting stakeholders
Number of studies completed as part of the 2013-2015 round of
evaluations
Analytic pipeline proposal for future work to address new
frontiers/special topics of concern to PSF stakeholders
Capacity-building action plan for Indonesian research partners
developed and implemented
Short-course training modules – focused on technical details in
implementing qualitative and/or quantitative social science research –
delivered
Executing Agency: PSF Secretariat/World Bank
Estimated Budget: US$ 4,750,000
Estimated
Duration: 32 months (covering activities to December 31, 2015)
Page 2 of 25
Geographic
Coverage:
National
Implementation
Arrangements:
PSF will use specified funding from the PSF Trust Fund to issue
contracts to qualified research organizations and/or consultants to carry
out the analytic works.
PSF Secretariat
Points of Contact:
Natasha Hayward, Senior Social Development Specialist
other local administrative or government spheres; impacts on households in less poor kecamatan
are limited; both PNPM Rural and Urban have not been able to address the constraints to long-
term, productive livelihoods in all households and communities; and PNPM Rural’s massive
scale-up (over an abbreviated period) has produced a number of challenges in both scale and
scope in the managerial, quality control, and operations fields.
PSF analytic work: what difference does it make?
Over the years, findings from these studies and the research ‘evidence base’ have:
• Informed and influenced policy dialogue and decisions
Presidential decision to scale up KDP as PNPM National in 2007; and subsequent commitment by present Administration to sustain PNPM as its core vehicle for community-based poverty reduction, at national scale through 2015
PNPM Roadmap – data and evidence contributed to ongoing Working Groups deciding on how to customise/tailor PNPM implementation and block grant formula allocation, according to ‘need’ (based on evidence of where most impact is seen; and utilizing census findings)
RUU Desa – evidence (e.g., on critical role of accountability mechanisms at local level) provided to inform/influence content of draft village law currently being prepared for parliamentary approval;
Calculation of a supply-readiness index for basic health and education services, which when combined with data on use and access to services is being used by GoI to prioritize areas for PNPM Generasi expansion for 2014-2019
• Influenced/led to innovative pilots
PNPM Peduli’s innovative model of partnering with CSOs to better target and empower the most marginalized groups
Creative Communities’ approach of utilizing and reinterpreting traditional media and forms of cultural expression to reinvigorate and promote participation and voice in local level decision-making
Bappenas intention, with PMD/PU, to pilot a livelihoods-focused PNPM model, addressing jobs and income under MP3KI
Generasi’s testing of an ECD indicator in three districts and the upcoming introduction of a "nutrition sensitive" indicator beginning in 2014
• Contributed to operational adjustments and lessons
Mainstreaming of Generasi pilot’s performance-based design to all project locations; PNPM Rural’s policy of 80% audit coverage Generasi’s scale up locations determined according to evidence of where the program is most
effective Operational guideline that one woman member is mandatory on the local Procurement committee Facilitator workload analysis to inform assignment of facilitator roles under PNPM Roadmap plans
Page 4 of 25
4. The GoI has committed to supporting PNPM through 2014 and has developed a
“Roadmap” for the initiative to provide guidance for that period and beyond. A major pillar in
the GoI’s Roadmap is the mainstreaming and integration of PNPM operating principles into
local government, administrative, and service delivery operations. Quality of service
delivery and accountability relationships are critical challenges especially in Indonesia’s ongoing
decentralization process and the shifting equilibrium between central and local authority and
responsibility. Currently, Indonesia, while a lower middle-income country, has poor public
service delivery standards (and outcomes) far below regional peers, income peers, and even
some highly disadvantaged low-income countries. In addition to these challenges, issues are
emerging in the practice of governance in the national-scale PNPM program and in demands for
attention to economic opportunities and livelihoods development for the poor via the PNPM
platform, as described in the Masterplan for the Acceleration and Expansion of Poverty
Reduction in Indonesia (MP3KI) document.
5. The PSF M&E framework for PNPM (and associated pilots and initiatives) therefore
supports the GoI’s Roadmap and other commitments to maximize community-led development
at the local level by linking operational experience and local-learning with applied research
and evaluation, all in the service of a knowledge base of best-practice solutions for effective
CDD and community-led programs in Indonesia. The M&E team’s analytical agenda will
focus on research and solutions that aim to measure and increase such programs’ pro-poor
impacts and facilitate a better understanding of social and institutional dynamics at the local level
and their influence on community empowerment and development strategies and policies.
6. This top-up funding proposal covers a limited set of activities from this larger,
overall agenda. This request covers quantitative and qualitative evaluation pieces that have
been identified as crucial for effective evaluation of present PNPM/CDD programs. It was
agreed with development partners that this evaluation work should be prioritized for action while
allowing the special topics agenda to crystallise based on donor and GOI priorities (e.g., as plans
for increased focus on rural livelihoods are articulated; as plans for follow up to the PNPM
Urban evaluations are agreed, etc). See Section IV below for further detail on the subset of
activities covered by this top-up funding proposal.
7. A specific exercise will therefore be undertaken in the second half of 2013 with all
stakeholders, to assess needs and to critically review and redefine future frontiers for PSF’s
analytic pipeline going forward, beyond these core evaluation commitments and in the light of
the new PSF strategy, under preparation in May-June 2013. This review will result in a revised
set of hypotheses and an overarching Analytic Framework for 2013-2018, for which additional
resources may be requested in due time to cover any additional resulting plans.
8. The Framework will be reviewed annually as a reference from which to prioritise and
address agreed questions and related studies. It is anticipated that this ‘review and redefinition’
exercise will take stock of the key trends and changes that have occurred in Indonesia’s
demographic, political and socio-economic landscape over the last 15 years, as also embodied in
the Government’s forward looking Roadmap for PNPM and MP3KI Masterplan for Poverty
Alleciation. At the same time as consulting and assessing stakeholders’ latest priorities, the
Page 5 of 25
review would also critically revisit some of the influential conceptual frameworks used as a
reference point in the evolution of the PNPM platform and pilots, identifying any related analytic
gaps and future horizons for exploration in the portfolio (e.g., livelihoods development; frontline
service delivery, etc).
9. The PSF M&E agenda, developed in collaboration with GoI, has consistently placed
equal emphasis on rigorous impact evaluations and quantitative studies as well as detailed
case studies, longitudinal qualitative work, and special studies. The 2013-2015 core agenda
approaches the construction of the evidence base in the same way, and the M&E group and
partners believe quality and depth of research – rather than methodology – are paramount. For
that reason, the push to include more national partners will be accompanied by an emphasis on
quality control tailored to a research partner’s needs and which evolves as skills, capacities, and
personnel evolve.
10. More joint outputs with national research bodies and GoI task forces will be
pursued; enhancing partnerships with both international and Indonesian universities and think
tanks; and active collaboration with Indonesian research firms and groups, all of whom are
expected to absorb more analytical and research services and products in the years to come.
II. Project Development Objectives
11. The M&E agenda was originally developed to: (1) provide stakeholders with empirical
data on program impacts and outcomes in PNPM communities; (2) plan and conduct
research in special topics of concern to those in PNPM operations as well as stakeholders
more generally; (3) support capacity building within, and skill transfer to, Indonesian
research organizations. These objectives remain relevant and it is expected that they will foster
an environment which encourages research in social development and/or community-led
development in Indonesia.
III. Project Description
12. This Project consists of the following components:
i. Monitoring and evaluation (M&E) activities to provide stakeholders with empirical
data regarding the results and impact of PNPM Mandiri;
ii. Research in greater depth undertaken regarding special topics of concern to PNPM
and the social development field in Indonesia;
iii. Capacity building to enhance ability of Indonesian social science research
organizations to conduct M&E and special studies.
13. PSF’s current strategy1 formulates four action-oriented pillars to guide operational and
analytic activities: continued poverty impact, improved accountability in service delivery,
1 Note that the 2012 PSF Strategy is undergoing revision and an updated Strategy will be presented for JMC
approval in July 2013. The present strategy, and its associated Analytic Framework, has served as the reference
point for the core analytic questions covered in this M&E proposal. As noted in paragraph 8, a review and
Page 6 of 25
social justice and inclusion, and a supportive enabling environment. These pillars also serve
as the organizational frame behind the M&E analytic agenda contained in this proposal (see
Table 1 below and Annex 2 for further detail on this organizing framework, associated research
questions and resulting evaluation topics).
14. The main organizing questions are:
a. Continued poverty impact: How, why and where does PNPM reduce poverty and how
can poverty-reduction impacts and rural livelihoods development be enhanced?
b. Improved accountability in service delivery: In what ways and under what conditions
can community participation improve local governance and service delivery for all?
c. Social Justice and Inclusion: What does PNPM need in order to be more inclusive and
enhance opportunities and empowerment of women, the very poor, and the marginalized?
d. Supportive Enabling Environment: How can the proposed reforms relating to PNPM and
other poverty-reduction or pro-poor growth initiatives be supported by and coordinated
through an appropriate combination of policies, institutions and operational systems and
capacities?
Table 1 – Organizing Framework and Guiding Research/Evaluation Questions
i. Continued poverty impact: How, why and where does PNPM Rural reduce poverty?
1. Related Research Questions Policy/program relevance
a. How are local economies, productivity and
rural livelihoods affected by PNPM
infrastructure/investments?
How to adapt PNPM for further livelihoods
development? In what contexts does PNPM work
better? How should the program be geographically
targeted?
b. Is PNPM as an instrument fulfilling its stated
social and economic objectives? What is the
role of Cluster 2, visa-vis Clusters 1 and 3?
Can PNPM support GoI’s social protection
efforts?
Crystallizing PNPM’s role and efficacy in the logic
of GoI’s poverty strategy, mediating between C1
and C3, as a platform for accountability in delivery
of programs/services.
c. What is PNPM’s effectiveness in Indonesia’s
most disadvantaged/poorest regions? (see also
Q2.) (e.g., evaluation shows greatest poverty
impact in poorer areas/households, but
implementation challenges and bottlenecks
are greatest in poorest and most remote areas)
Optimizing PNPM effectiveness where needs and
challenges are greatest. Should PNPM design,
activities and indicators be adjusted for
regional/socio-cultural variation?
d. Which aspects or features of PNPM are
performing better? How does the PNPM
delivery mechanism for particular
outputs/investments compare to other
(government) options? What poverty
reduction can be attributed to PNPM?
Identifying the most effective parts of the PNPM
process/delivery mechanism. Continued justification
of program as core of Cluster 2.
2. Improved accountability in service delivery: How does community participation and
empowerment transform local governance and service delivery?
redefinition exercise will take place later in 2013, to update an Analytic Framework for 2013-2018, ensuring
consistence with the updated PSF Strategy 2013, and including any special topics/research questions or thematic
studies not covered in this ‘core’ proposal.
Page 7 of 25
2. Related Research Questions Policy/program relevance
a. What are local understandings, pathways and
practices of accountability in Indonesia’s
diverse contexts?
Identifying effective mechanisms to foster better
downward accountability and transparency, including
channels for recourse and grievance redress.
b. Why do/don’t PNPM-specific governance
processes translate into other broader local
governance/service delivery arenas? Why is
it hard to institutionalize accountability and
participation? What impacts do broader
processes of decentralization and politics
have on space for empowerment through
PNPM?
Identification of conditions for positive synergies
between PNPM and village government.
Identification of PNPM’s limits, and wider pressure
points beyond the program. Identifying and
rebalancing asymmetries of information in the rural
space.
c. What can the PNPM platform do and how
far can it be ‘exploited’? Where are
communities most effective and what are
their limitations?
Identifying key community-driven
practices/institutions/functions to be institutionalized
within local planning mechanisms. What roles can
communities play in co-production/co-delivery of
services?
d. What encourages sectoral and LG
partnership?
Identifying proper incentives. Informing debate on
what responsive governments can deliver.
3. Social justice and inclusion: How can the PNPM platform be more inclusive and foster
empowerment of women, the poor and marginalised?
3. Related Research Questions Policy/program relevance
a. Do elites capture or contribute to PNPM?
Where, how and why?
Identifying design elements that will lower capture or
effectively utilize elite capacities
b. How well does PNPM target the poor and
marginalised? Why can’t some groups
participate? What can we learn from other
more targeted delivery mechanisms and
modalities (Pekka, Peduli, DPOW, etc)?
Identifying design/implementation reforms that will
promote more inclusion/greater participation and
support social justice for typically excluded groups.
c. Does PNPM address the needs (institutional,
social, financial) of poor rural women?
What can we learn from other delivery
mechanisms and modalities (Pekka, Peduli,
Justic, etc)?
How can targeting of women be improved? What
should be goal and design of RLF?
d. What is PNPM’s effectiveness in
Indonesia’s most disadvantaged/poorest
regions? (see also Q1)
Should PNPM design, activities and indicators be
adjusted for regional/socio-cultural variation?
4. A supportive enabling environment: How can proposed reforms be supported by and
coordinated through an appropriate combination of policies, institutions and operational systems
and capacities?
4. Related Research Questions Policy/program relevance
a. What are the PNPM core functions, systems,
principles to be sustained under any vision of
more integrated local and community driven
development?
Determination and agreement on those ‘benefits’
(institutions, practices, etc) to be sustained under
the future vision
b. What are the institutional and policy
arrangements (at central and local levels) that
would support proposed reforms? What are the
How do existing formal/informal local institutions
contribute to en enabling environment for inclusive
development and local governance? Identifying a
Page 8 of 25
institutional impediments to sustainable local
development? (see also 2(b)) How do non-
governmental actors (NGOs, CSOs, staff)
mediate/support PNPM’s effects?
joint framework for village level institutions (inc.
options for facilitation) to support inclusive local
development.
c. How to institutionalize technical capacity in
M&E (especially qualitative and formal IE
methodologies) outside of PSF?
Independent research institutes/university
department/think-tanks with sufficient capacity for
applied and longer-term social analysis to inform
GoI’s poverty programming.
15. Guided by the framework and research questions above and in order to provide policy-
relevant evidence, this Project directly supports evaluation plans linked to key Cluster 2 projects:
(a) PNPM Rural and the related PNPM Mandiri Roadmap; and, (b) PNPM Generasi. The main
analytical projects are:
a. PNPM Rural and Roadmap Evaluations:
Economic Impact Simulation – This exercise will demonstrate the shape and size of
the benefits PNPM Rural brings to the Indonesian economy and rural livelihoods. It
will compare the “PNPM way” of addressing local infrastructure and productive
capital needs to a simulation of the next-best alternative for addressing those same
needs. By cataloguing the local productivity gains, contributions to local economies
and livelihoods, and cost savings for the central government, the Economic Impact
Simulation will be a comprehensive summary of the measurable financial advantages
of PNPM principles and an identification of the most effective parts of the PNPM
process/delivery mechanism in supporting local economic gains.
Sentinel Villages – The aim of the Sentinel Villages project is to better understand the
changing nature of community experience and satisfaction with, as well as desires for,
the PNPM program and community members’ participation in the program.
Understanding the way these experiences evolve will in turn lead (in real time) to
suggestions for changes in program implementation. The methodology will combine
repeated qualitative work and analysis over a number of years with a quantitative
analysis based on those same longitudinal qualitative records. Research foci will
include the changing nature of participation, social capital, and the evolution of
democracy at the local level and its impacts on PNPM’s value as a community
solution. Results will directly support PNPM policy and Roadmap elaboration,
especially those policies focused on engaging local government effectively in poverty
reduction programs and other pro-poor development initiatives.
Accountability initiative: This project will take further the question on what enables
or constrains PNPM-type good governance and accountability practices to be taken
up more widely within the community or local government or administration, and
how this relates to service delivery/local government performance at the local level.
Based on global and local experience, a series of propositions for testing key
hypotheses linked to effective triggers for improved accountability at the local level
are being developed. This piece will be linked to the PNPM Roadmap and wider local
governance initiatives under PSF which may include: the planned Enhanced
Page 9 of 25
Empowerment Experiment (EEE) pilot which will test mechanisms for improving
participation and accountability; the Community Legal Empowerment project (in
partnership with Pekka) and the overall ‘frontline service delivery’ package supported
by Ausaid.
Revolving Loan Fund (RLF) Expenditure Tracking exercise – The Revolving Loan
Fund initiative has delivered business-oriented credits to every community with
PNPM block grants. However, very little is yet known about what local enterprises
or households these credits supported. The GoI would like the RLF initiative to be an
effective vehicle for giving both savings and credit access to marginalized households
and communities as part of its rural livelihoods strategy, but has a limited evidence or
knowledge base to consult for RLF-centered policy deliberations. This tracking
exercise is seen as the first building block in the construction of an RLF-centered
evidence base and as a complement to the ongoing RLF pilot exercise (and associated
evaluation). Primary quantitative research questions will include: What are the
characteristics of an RLF beneficiary? What are the characteristics of an RLF-group
manager? To what uses were RLF credits put? How did beneficiaries treat RLF
credits (i.e, as actual credits or something closer to a consumption-smoothing
vehicle)? The evidence gathered can provide preliminary policy direction as well as
provide impetus for the next round of analytical and research activities focused on the
RLF initiative.
Gender, livelihoods and empowerment: Having supported a baseline evaluation for
the PEKKA program – a community-driven initiative targeting poor widows and
female-headed households – PSF will continue its support to PEKKA’s evaluation by
assisting PEKKA with analysis of available datasets to determine the program’s
quantitative impact, and preparing a program of associated qualitative work to
understand what is behind PEKKA successes and challenges, learning lessons of
importance on gender, livelihoods and empowerment of key relevance to the PNPM
approach to community empowerment and poverty reduction.
b. PNPM Generasi Evaluation:
Quantitative Impact Evaluation: Building on the previous Generasi impact evaluation
(over the years 2007-2010), and with the benefit of controls largely having been
maintained in the original pilot kabupaten, a feasibility study is planned to ascertain
whether a repeat fourth round of the IE survey instruments may be possible. This is
planned to examine impacts and results from an enhanced Generasi design in new
provinces and to continue to generate longitudinal panel data on Generasi's impacts in
areas which continue under the program. With MCC also fielding a formal Impact
Evaluation in their 3 provinces, the team will also provide technical and coordination
inputs to that exercise.
Qualitative studies: the Generasi team is planning qualitative pieces in order to
complement any quantitative evaluation and better understand how accountability
between community members and service providers is created and sustained. These
studies, which the PSF M&E agenda will also support, may include: (i) lesson
learning case studies detailing the interactions between community members and
Page 10 of 25
frontline service providers; (ii) in those areas where Generasi will phase out in 2013,
a prospective study summarizing the sustainability of community capacity that was
built up through six years of participation in PNPM Generasi.
16. Continuing best practices from the past, this Project will use a number of different
approaches to answer the questions framed by these core evaluation pieces. Methods that
may be used by the Project include:
Impact evaluations to assess the effects of operations;
Experimental interventions to test and empirically assess how altering elements
of programs can enhance outcomes;
Qualitative and ethnographic studies to explore in greater depth the factors
explaining program performance or other presenting issues of concern;
Quantitative and spatial analysis to understand and assess sources of variation;
Stocktaking and longitudinal reviews on specific themes and program areas.
17. The Project will also develop mechanisms and initiatives to strengthen capacity in
local social research organizations to deliver high quality analytical work (see
implementation arrangements below). The core M&E team is relatively lean – it is a group of 4
fulltime specialists – so the team will link study and research implementation with a capacity
building strategy to enhance the ability of Indonesian social science research organizations
(including both universities and independent think tanks) to conduct M&E and other special
studies. To date, this has largely been addressed through an implicit partnership mode, with all
studies executed alongside a local research institution, building methodological training goals
and field-level quality assurance into each activity. This modality has demonstrated its limits, so
the team is planning a more explicit set of capacity-building and skill-transfer activities,
primarily to address local research institutes’ capacities for qualitative methodologies and
quantitative impact evaluation skills.
18. Coordination of plans and dissemination of analytic activities across interested
Cluster 2 partners – and beyond – will be a priority. A primary point of reference for sharing
of plans, with opportunity for cross-coordination and inputs, and as a venue for dissemination of
findings will be the PSF JMC. In addition, a further mechanism (e.g., an Analytic working
group) for practical coordination purposes will be agreed with the Technical Committee of the
JMC, in order to function as a forum with regular meetings and virtual opportunities for sharing
and discussion.
19. For dissemination, in addition to the more conventional channels, such as presentations to
key stakeholders/institutions and publication of related reports and digestible summaries, all
materials will be made available via the PSF website. The team recognizes however, that such
channels remain limited and so, for major studies, or sets of studies, tailored dissemination plans
will be prepared and the JMC and analytic coordination group will serve as advisory boards to
provide inputs on new audiences and channels for dissemination. It is also proposed that early
drafts of Concept Notes, draft reports, presentations, etc are shared for comments with members
of the working group, to give an opportunity for inputs and reactions throughout the process and
not simply at the conclusion of each study process.
Page 11 of 25
IV. Project Outcomes and Deliverables
20. The results indicators in the table below were developed specifically to correspond to
activities covered by this top-up funding request. For more detail on the links between the body
of evidence so far accumulated from completed PSF M&E research and the ongoing 2013-2015
program, also see Annex 1.
PDO Level Result
Indicators
Unit
of
Meas
ure
Base
line
2012
Cumulative
Target
Annual Targets (cumulative) Data Source/
Methodology 2013 2014 2015
Project Development Objective(s): (i) to provide stakeholders with empirical data regarding the results and
impact of PNPM Mandiri; (ii) conduct research in greater depth regarding special topics of concern to PNPM
and the social development field in Indonesia; and, (iii) support capacity building efforts for Indonesian social
science research organizations
(i) To provide stakeholders with empirical data regarding the results and impact of PNPM Mandiri
Data and findings
used to improve
implementation and
enhance impact in
existing and new
programs. N/A N/A
Findings
incorporated in
programs/projects
’ design and
implementation
documents
Findings
of
completed
studies
available
and
adopted by
relevant
task teams
Findings
of
completed
studies
available
and
adopted
by
relevant
task teams
Findings
of
completed
studies
available
and
adopted
by
relevant
task teams
Evaluations/stu
dy reports;
project
documents;
task team
feedback
Evidence and
recommendations
regarding sustaining
poverty-reduction
and pro-poor growth
impacts of CDD
programs delivered
to PNPM
stakeholders
(Continued Poverty
Impact)
N/A N/A
Policy-focused
recommendations
identified and
delivered to
stakeholders
Rural
Infrastruct
ure
Census;
PNPM;
Respek
Evaluation
PNPM
Rural
Impact
Simulatio
n
Sentinel
Villages
report
Evaluations,
analysis,
reports
Evidence and
recommendations
regarding PNPM,
local economy and
livelihoods delivered
to stakeholders
(Continued Poverty
Impact)
N/A N/A
Policy-focused
recommendations
identified and
delivered to
stakeholders
PNPM
Respek
Evaluation
PNPM
Rural
Impact
Simulatio
n; RLF
tracking
PEKKA
evaluation
Evaluations,
analysis,
reports
Mapping of
community groups’
contributions to
N/A N/A
Graphic &
summary guide to
accountability
-
PNPM-
Generasi
Eval.
Accountab
ility
Evaluations,
analysis,
Page 12 of 25
accountability in
achievement of
minimum standards
service delivery,
delivered to
stakeholders
(Improved
Accountability in
Service Delivery)
relationships at
the desa-or-
below- level
initiative reports
Evidence regarding
local constraints to
adopting PNPM
governance practices
delivered to
practitioners’ forums
(Improved
Accountability in
Service Delivery)
N/A N/A
Summary note
describing how,
where, and why
community
groups can
successfully
change operating
procedures to
their benefit
LLI3
report;
Sentinal
Villages
report
Accountab
ility
initiative;
Sentinal
Villages
report
Evaluations,
analysis,
reports
Recommendations
regarding strategies
for increasing
inclusion and
outreach to
marginalized
households and
groups delivered
(Social Justice and
Inclusion)
N/A N/A
Policy-focused
recommendations
identified and
delivered to
stakeholders
- PEKKA
evaluation
PEKKA
evaluation
Evaluations,
analysis,
reports
Research activities,
products, and
services planned for
and developed in
collaboration with
PNPM/Cluster 2
policy-setting
stakeholders
% of
produ
cts
N/A
100% of studies
developed with
stakeholder input
100% 100% 100%
Preparatory
meetings;
advisory board
memberships
Number of studies
completed as part of
the 2013-2015 round
of evaluations2
# of
Evalu
ations
/Studi
es
0 9 3 6 9
Completed
reports
published on
website
(ii) To conduct research in greater depth regarding special topics of concern to PNPM and the social
development field in Indonesia
Analytic pipeline
proposal for future
work to address new
frontiers/special
topics of concern to
N/A N/A
Report
summarising
critical lessons,
analytic gaps
and key future
Review
and
redefinitio
n exercise
completed
Consultations
and review
report
2 Including LLI3 and Respek evaluations financed under the existing M&E TF allocation
Page 13 of 25
PSF stakeholders research
questions/horiz
ons for PSF
(iii) To support capacity building efforts for Indonesian social science research organizations (Supportive
Enabling Environment)
Capacity-building
action plan for
Indonesian research
partners developed,
endorsed by JMC
members, and
implemented
NA NA
Indonesian
research
partners
capacities’
increased
Capacity
building
action-
plan
endorsed
by JMC
members
Capacity
building
action-
plan
resourced
and
implement
ation
started
Capacity
building
plan
implement
ed
Procurement
contracts,
progress
reports
Short-course training
modules – focused
on technical details
in implementing
qualitative and/or
quantitative social
science research –
delivered
# of
cours
es
held
N/A
Ensuring a
supply of basic
social science
research skills
and upgrading
existing
embodied
capacity
Course
designed
1 course
held
2 course
held
Training
modules and
curricula
V. Implementation Arrangements
21. The Project will be led by the PSF M&E team who will be in charge of directly
developing or overseeing development of the conceptual framework for each piece of analytical
work, leading the implementation of the works in partnership with local research institutes, and
providing quality control of data collection, analysis and reporting. With a renewed focus on
partnership, the team will also explore some more innovative delivery models for discrete studies
and be pro-actively engaging relevant policy-level stakeholders in leadership of discrete pieces
or in contributing to concept development, review and dissemination stages for each piece.
22. While capitalizing on PSF in-house team strengths, the team will work in
partnership with local research institutes and other parties to carry out the Project. This
will be part of the team’s capacity building strategy. The overall approach of this strategy is
intended to be consistent with the AusAID-financed Revitalising Knowledge Sector strategy in
Indonesia – a longer-term effort that will address both individual and organizational capacity in
social science research capacity – and to build on existing collaboration with the World Bank’s
PREM Poverty team. The PSF M&E effort, in delivery of this proposal, is intentionally selective
in scope, in acknowledgement of the skill-set, capacity and other commitments of the team. Key
elements to be addressed in the immediate phase include:
i. Short-course training modules in qualitative methods: delivered in partnership with
local resource institutions, before handing over full responsibility to those partners.
The goal of the design and offering of such short-course trainings would be to ensure
basic analytical skills, familiarity with up to date theoretical and methodological
approaches, and ability to use these, primarily in applied qualitative research among
Page 14 of 25
those already engaged in social science research, but with need to enhance staff skills.
A potential module/partnership for quantitative IE would also be explored. The mode
of design and delivery would involve a core partnership between PSF, PREM and
identified local partners (currently discussing with Akatiga and SMERU), and may
involve potential international partners in a related ‘practice network’ (see para. 23
below), once endorsed by the JMC.
ii. Organisational strengthening. While the Asia Foundation has trialed a holistic
organizational approach to strengthening of local CSOs (under the Knowledge Sector
pilot), the PSF M&E team does not have a comparative advantage in overall
organizational capacity review/support, so, under this proposal, is planning an approach
more in keeping with the team’s technical skills and needs. This would involve
targeting selected local research institutes with undeveloped potential (the so-called ‘2nd
tier’ of independent research institutes), particularly in qualitative methodologies,
through potential developmental assignments of selected individual staff within PSF
over a 12-24 month period. Current thinking would be to mobilize a team of
individuals selected from a number of local institutes, to come on ‘assignment’ to PSF
and participate as members of a dedicated qualitative support team for an identified
period, prior to returning to their ‘parent’ institutions – with a commitment to then
remain with those ‘parent’ institutions for a defined period.
iii. Enhanced partnership model for discrete studies (e.g., Papua UNCEN/UNIPA
model). Taking into account a study’s geographic or thematic focus, the team will
engage related university bodies/faculty and other CSO representatives to sit on a study
oversight/review panel as Senior Advisors. These key faculty will contribute to
concept design, participate in methodological training exercises alongside PSF team
members and the hired local research institution; and assist in recruiting a number of
their students on ‘learning’ field assignments as Junior Researchers alongside teams in
the field. These Junior Researchers would have discrete outputs that would not directly
impact the overall study output, but would introduce the students to the processes of
data gathering, analysis and presentation of findings. They would be mentored by the
Senior Advisors and exposed, through the process, to methodological training; the
practice and experience of working with a professional (local) research body and
interaction with the PSF team
23. Future partnerships and innovations in delivery: In parallel to delivery of this key
phase of evaluations, the team is exploring options for developing additional critical partnerships,
with both local and international partners, to build networks for a collaborative research and
capacity building program in Indonesia, taking advantage of entry points offered by the
Knowledge Sector initiative and with a focus on strengthening of qualitative and applied
ethnographic methodologies3. The team may also explore the potential for a contractual
partnership with an experienced firm who may take on leadership and delivery of a discrete
‘package’ of the existing work covered in this proposal, with guidance and quality assurance
from the M&E team.
3 A separate Note is available describing these plans.
Page 15 of 25
VI. Estimated Budget
24. The Project team is currently seeking approval of US$4,750,000 for the two-year project
(See below for a breakdown and Annex 3 for a breakdown of specific evaluation/study budgets).
Category US$
(1) Evaluations/studies $3,000,000
(2) Capacity building for local research institutes $350,000
(3) Supervision and implementation support – including
staffing, field oversight $1,400,000
ALLOCATION REQUESTED $4,750,000
Page 16 of 25
Annex 1: PSF M&E Program, Studies Completed or Ongoing in 2010-12
Table 1: Completed Evaluations/Studies
Evaluations/Studies Methodology Main Findings
1 PNPM-Rural Impact
Evaluation
Quantitative
(propensity score
matching);
endline 2010
Real per capita consumption gains of 9.1 % for households in PNPM-Rural
areas
The gains were higher for poor households and poor sub-districts (11.8% and
12.7% respectively)
Households in PNPM areas were 2.1% more likely to move out of poverty
Access to health services increased 5.1% for households in PNPM areas
2 PNPM-Rural Qualitative
Impact Evaluation
Qualitative,
multi-years (2007
and 2010)
Participation, transparency and accountability were strong within the
program, especially at the sub-district and village levels
PNPM is most effective at reducing poverty and impacting poor households
when the needs of the poor are aligned with those of the community
3 PNPM-Rural Marginalized
Groups Study
Qualitative
(2010)
Marginalized groups have limited participation in PNPM meetings which
were still dominated by elites and interest groups
Although facilitation/project procedures have often led to increased
participation rates, it has not encouraged active or influential participation of
marginalized groups in the development of proposals
Officials and leaders of interest groups (religious and traditional elites) still
possess the greatest influence over which proposals are developed and
selected, therefore reducing opportunities for marginalized groups to impact
Page 17 of 25
Evaluations/Studies Methodology Main Findings
decision-making on the use of project resources
4 Village Capacity in
Maintaining Infrastructure
Study: Evidence from Rural
Indonesia
Quantitative
(repeated surveys
every quarter in a
year to capture
cyclical income
fluctuation, 2010)
Many communities still found it difficult to maintain existing infrastructure
projects
The cost of maintaining infrastructure was up to 2.8 percent of a household’s
total consumption – although small is likely burdensome for poor households
Although villagers are willing to pay for the maintenance of infrastructure,
the amount communities are willing to pay does not cover all infrastructure
maintenance needs
The community’s willingness to pay is strongly influenced by the direct
impact of the infrastructure on individual households and institutional
responsiveness in terms of complaint handling
5 PNPM Generasi Impact
Evaluation
Quantitative
(randomized
control trial,
baseline 2007,
endline 2010)
Significant impacts on all 12 indicators
The strongest improvements among the health indicators were in the
frequency of weight checks for young children
The improvement in education indicators was most notable in the increased
school participation rate among the primary school-age group
In terms of long-term impact, there was about 10 percent decrease in
malnutrition compared with control areas
6 PNPM RESPEK
Infrastructure and
Community Organization
Capacity Evaluation
Mixed-methods
(2011)
All infrastructure samples evaluated are of good to moderate quality –
significant achievement considering huge implementation challenges in Papua
and West Papua
67% of the infrastructure built were utilized by the community
Page 18 of 25
Evaluations/Studies Methodology Main Findings
PNPM RESPEK improved administrative capacity of local facilitators but not
the facilitation capability
7 Rapid Assessment of
Women’s Participation in
PNPM
Qualitative
(2012)
Quality of women’s participation was mixed in study areas and is still not
maximized
Women’s involvement in the project cycle is still low beyond the initial
project stages of socialization and needs assessment
Women’s proposals that are funded are usually infrastructure, rather than
e.g., capacity building and trainings that are requested when the local actors
do a more in-depth analysis of women’s needs
Local initiatives and strategies around women’s participation existed but yet
to be integrated into more general local decision making processes
8 Local Level Governance
Review
Qualitative/
Action Research
(2012)
The state of local governance in PNPM is mixed: it is still remarkably strong,
but faces problems.
Participation rates are still mostly high, but the quality of participation has
declined in some places.
There are weaknesses in transparency and information sharing: performance
of accountability mechanisms in the program varies and the incidence of
serious corruption is up.
Despite this, the ‘foundations’ of PNPM are still strong. These foundations,
though, are being eroded by serious pressure from ‘higher-up’ problems
related to the broader governance environment, changes in the program
design, and problems with implementation and management.
PNPM must address these program design, implementation and management
issues to avoid undoing its years of good work at community level.
9 Rate of Return Analysis
(EIRR) of PNPM-Rural
Quantitative
(economic
analysis), small
Similar with 2005 results (Torrens, 2005), the EIRR varied among sub-
projects with median value of 30 – 50%
The average general income multiplier is 1.3
Page 19 of 25
Evaluations/Studies Methodology Main Findings
Infrastructure Sub-Projects scale study in 20
villages assessing
48 sub-projects
using similar
methodology as
in 2005 exercise
Sub-projects are generally 25 – 30% cheaper than project built using typical
local government contractor
10 Village Infrastructure
Census
Quantitative
(village level
census) –
piggybacking
PODES 2011
Comprehensive data on basic infrastructure availability and quality (including
health and education facilities) in nearly all urban and rural villages in
Indonesia (over 76,000 villages).
A consistent picture of geographic variation in the supply of basic
infrastructure across Indonesia. In general, the island of Java and the province
of Bali perform best, while local needs for investment still exist in these
regions, and particularly so in the provinces of Jawa Barat and Banten.
The largest gaps in infrastructure supply readiness are found for the Papua
region, the Maluku islands, NTT, as well as for the interior of Kalimantan.
Data has been used to create a supply readiness index for health and
education (to support PNPM Generasi in location selection/targeting).
Further analysis will be done to calculate financing gaps (pending data
availability on costs) and to provide more information for PNPM-Rural in
terms of location selection and allocation of block-grants
Table 2: Ongoing Evaluations/Studies: 2012
No Name of Study Description of Study Methodology Expected Timeline
for Result
1 Local Level Observes changes in local organizational capacity and
social capital (embodied in participation in community-
Mixed-methods
(longitudinal study,
First draft report
Page 20 of 25
No Name of Study Description of Study Methodology Expected Timeline
for Result
Institutions 3 (LLI3) based organizations); links changes in such capacity to
shifts in community-members’ influence over
development decision-making, project implementation,
and service provision at the district and community levels.
1st one in 1996 and
the 2nd
in 2001)
expected Q1 2013
2 Incidence of
Household Benefits
Determining who actually benefits from PNPM as well as
understanding community’s perspective of PNPM
including its uses and its capacity to target the poor and
vulnerable in particular.
Quantitative Report expected Q1
2013
3 Integration of
Community-based
Poverty Reduction
Programs at
Community Level
Examining integration of CDD programs at community
level and observing local community capacity to manage
diverse development programs; lessons learned will be
used as inputs to the development of integration strategy
in PNPM Roadmap.
Qualitative First draft report
expected Q1 2013
4 Papua Operational
Research and
Evaluation
Several analytic pieces to be done related to PNPM
RESPEK and conflict issues in Papua: (i) a qualitative
‘beneficiary assessment’ to elevate voices from the field;
(ii) piggybacking SUSENAS with a specific PNPM
module to collect and benchmark quantitative
data/indicators from Papua and West Papua; and (iii)
operational mapping/location profiling in Papua.
Mixed-methods Concept note(s)
completed by Q4
2012
Page 21 of 25
Annex 2: Mapping of PSF 2013-2014 Studies to Organising/Research questions.
Research Question Policy/program relevance Ongoing or Proposed
Activity
Anticipated
timeline
1. Continued poverty impact: How, why and where does PNPM reduce poverty?
a. How are local
economies,
productivity and
livelihoods affected
by PNPM
infrastructure?
In what contexts does PNPM
work better? Should the
program be geographically
targeted? How to adapt
PNPM for further livelihoods
development?
i. PNPM 2012-
2014
Evaluation
(Econ Impact
Simulation)
i. 2013-14
b
.
Is PNPM as an
instrument fulfilling
its stated social and
economic objectives?
What is the role of
Cluster 2, vis a vis
Clusters 1 and 3?
Crystallising PNPM’s role
and efficacy in the logic of
GoI’s poverty strategy,
mediating between C1 and
C3.
i. PNPM 2012-14
Evaluation
(Econ Impact
Simulation and
possible 2013
Sedap repeat
round)
ii. Sentinel
Villages
iii. RLF Pilot
Evaluation
(tbc)
iv. Eval of
combined
effects of PKH
and Generasi in
areas of overlap
i. 2013-14
ii. 2013-14
iii. Tbc
iv. tbc
c. What is PNPM’s
effectiveness in
Indonesia’s most
disadvantaged/poores
t regions? (see also
Q2.) (e.g., evaluation
shows greatest
poverty impact in
poorer
areas/households, but
implementation
challenges and
bottlenecks are
greatest in poorest
and most remote
areas)
Optimising PNPM
effectiveness where needs and
challenges are greatest.
Should PNPM design,
activities and indicators be
adjusted for regional/socio-
cultural variation?
i. Papua PNPM
Rural/Respek
Qualitative
‘Voices’ study
ii. Papua
Rural/Respek
Quantitative
Evaluation
(SUSENAS
2013)
iii. PNPM 2012-14
Evaluation
i. 2012-13
ii. 2013
iii. 2013-14
d
.
Which aspects or
features of PNPM are
performing better?
How does the PNPM
Identifying the most effective
parts of the PNPM
process/delivery mechanism.
Continued justification of
i. Rural
infrastructure
census
ii. PNPM 2012-14
i. 2012
ii. 2013-14
iii. 2013-14
Page 22 of 25
Research Question Policy/program relevance Ongoing or Proposed
Activity
Anticipated
timeline
delivery mechanism
for particular
outputs/investments
compare to other
(government)
options? What
poverty reduction can
be attributed to
PNPM?
program as core of Cluster 2.
Evaluation
(Econ Impact
Simulation and
possible 2013
Sedap repeat
round)
iii. Sentinel
Villages
2. Improved accountability in service delivery: How does community participation transform
local governance and service delivery?
a. Does empowerment
improve governance?
Why do/don’t
PNPM-specific
governance processes
translate into other
arenas?
Catalysing adoption and
wider take up of PNPM
processes, beyond the
program sphere. Identifying
key
practices/institutions/function
s to be institutionalised within
local planning mechanisms.
i. Governance
impacts
study/experime
nt
ii. Sentinel
villages
iii. LLI3
i. 2013-14
ii. 2013-14
iii. 2012-13
b
.
What encourages
sectoral and LG
partnership? What
can the PNPM
platform do?
Identifying proper incentives.
Informing debate on what
responsive governments can
deliver. What roles can
communities play in co-
production/co-delivery of
services? How to rebalance
assymetries of information in
the rural space?
i. Integration
study
ii. Generasi Phase
2 Evaluation
(Quant and
Qual)
i. 2012-13
ii. IE: Late
2013
(baseline
); late
2014
(midline)
late 2016
(endline)
c. What are local
understandings,
pathways and
practices of
accountability?
Identifying effective
mechanisms to foster better
downward accountability and
transparency.
i. Governance
impacts
study/experime
nt
ii. Community
groups
stocktaking/ana
lysis
i. 2013-14
ii. 2012
3. Social justice and inclusion: How can the PNPM platform be more inclusive and foster
empowerment of women, the poor and marginalised?
Page 23 of 25
Research Question Policy/program relevance Ongoing or Proposed
Activity
Anticipated
timeline
a. Do elites capture or
contribute to PNPM?
Where, how and
why?
Identifying design elements
that will lower capture or
effectively utilise elite
capacities
i. Incidence of
Benefit
ii. Sentinel
villages
iii. Elites study
i. 2012
ii. 2013-14
iii. 2014
b
.
How well does
PNPM target the
poor? Why can’t
some groups
participate? What can
we learn from other
more targeted
delivery mechanisms
and modalities
(Pekka, Peduli, etc)?
Identifying
design/implementation
reforms that will promote
more inclusion/greater
participation.
i. Incidence of
Benefit
ii. Sentinel
villages
iii. Pekka
Evaluation
Phase 2 (Qual
and Quant)
iv. Elites study
i. 2012
ii. 2013-14
iii. 2013-14
iv. 2014
c. Does PNPM address
the needs
(institutional, social,
financial) of poor
rural women? What
can we learn from
other delivery
mechanisms and
modalities (Pekka,
Peduli, etc)?
How can targeting of women
be improved? What should be
goal and design of RLF?
i. Sentinel
villages
ii. Pekka Phase 2
(Qual and
Quant)
i. 2013-14
ii. 2013-14
d
.
What is PNPM’s
effectiveness in
Indonesia’s most
disadvantaged/poores
t regions?
Should PNPM design,
activities and indicators be
adjusted for regional/socio-
cultural variation?
See 1(d)
3. A supportive enabling environment: How can proposed reforms be supported through an
appropriate combination of policies, institutions and operational systems?
a. What are the PNPM
core functions,
systems, principles to
be sustained under
any vision of more
integrated local and
community driven
development?
Determination and agreement
on those ‘benefits’ (optimal
facilitation arrangements,
institutions, practices, etc) to
be sustained under the future
vision
i. Integration
study
ii. Community
groups
stocktaking/ana
lysis
iii. Community
case studies (w/
Generasi)
i. 2012-13
ii. 2012
iii. 2012
Page 24 of 25
Research Question Policy/program relevance Ongoing or Proposed
Activity
Anticipated
timeline
b
.
What are the
institutional and
policy arrangements
(at central and local
levels) that underpin
sustainable and
accountable local
development? What
are the institutional
impediments to
sustainable local
development?
How do existing
formal/informal local
institutions contribute to an
enabling environment for
inclusive development and
local governance? Identifying
a joint framework for village
level institutions to support
inclusive local development.
i. Integration
study
ii. LLI3
iii. Community
groups
stocktaking/ana
lysis
i. 2012-13
ii. 2012-13
iii. 2012
c. How to
institutionalise
technical capacity in
M&E outside of PSF?
Research
institutes/unis/thinktanks;
central government; local
government? Focus?
i. PSF M&E
Capacity
building
initiative
Page 25 of 25
Annex 3: Proposed Study Budget Breakdown
No Planned Evaluation/Study Year Budget Year Budget Year Budget Total Source
