ADB TA 4581 —Developing a National Result-Based
M &E System for Key Projects
09-2006
Prof. Xue Yan, in behalf of TA 4581
TA 4581 Project Objective
Strengthen capacity of Key Project Inspectorate Office which is under the Ministry of National Development and Reform Committee (KPIO) to develop result-based performance indicators of key national projects.
Background
• KPIO created 1998:– major responsibility to monitor and inspect key
national projects;– first inspections January 1999.
• The magnitude of KPIO’s – >10,000 “national key projects”;– ~1,600 projects inspected up to 2005;– 85 staff (Beijing), ~500 (provincial & municipal
KPIOs).
Background KPIO’ s emphasis to date is on project implementation:
financial control schedule achievement quality of outputs corruption detection
This TA will assist development of results-based indicators to supplement the current system, and raise knowledge and awareness of Results-based monitoring & Evaluation methods. A a more complete picture will be gained, to tell: if the original purpose has been achieved; how well the project was implemented; how to improve operation of the finished project; lessons for future projects.
Project Approach
Test results-based indicators through selected project case studies.
Train KPIO staff and government officials, using case studies.
Integrate selected indicators into KPIO system.
Results-based Monitoring and Evaluation Systems
Results Results The outputs, outcomes and impacts of a project.
Monitoring Monitoring – Monitoring aims to ensure the project is on the right track and
necessary adjustments are made to ensure successful outputs and outcomes. Monitoring also provides the basis for Evaluation, as it demonstrates project progress and points to possible adjustments to project plans and targets.
– Monitoring is continuous from the beginning of a project.
EvaluationEvaluation The systematic and objective assessment of an in-progress or
completed project. Includes assessment of its design, implementation and results.
Results-based Monitoring Model
Strategic goal
Inputs Activities Outputs
Impact
OutcomeNeed
The Model with the Project Cycle
Project Project Implementation Implementation
PhasePhase
Project Project Implementation Implementation
PhasePhase
Strategic goal战略目标
Inputs输入
Activities活动
Outputs提交物
Impact长期效应
Outcome直接结果
Need需求
Strategic goal战略目
Inputs输入
Activities活动
Outputs提交物
Impact长期效应
Outcome直接结果
Need需求
ConceptConcept
FeasibilityFeasibility
DesignDesign
OperationOperation
Bidding & Bidding & ContractingContracting
In the past, project monitoring focused on implementation phase: activities and outputs.
Now governments increasingly focus on outcomes and impacts, to demonstrate value of projects in an accountable & transparent system.
Systems vary from country to country but all developed countries and many developing countries are working in the direction of Result-Based performance monitoring and evaluation.
Inputs
Activities
Outputs
Impact
Outcomes
Project budget.
Major activities carried out to create the project deliverables.
Physical deliverables of the project.
Statement of the long-term or strategic objectives.
Information on the expected direct results of the project being completed and put into operation.
The Framework
The framework summarizes how the project will achieve results by converting inputs to outputs, and how outputs are expected to give the desired results.
R-BR-B(addition)(addition)
NakprosesNakproses(current)(current)
Results-Based Development
Strategic goal
Inputs Activities Outputs
Impact
OutcomeNeed
The interesting parties
Ministry Department KPIOProject sponsor
Optimize performance Minimize waste Public accountability Good governancePerformance culture Benchmarking
State Council FinanceNDRCMinistry Critically evaluate benefits
Prioritize future projectsGood governance
Outputs
Impact
Outcome
Ministry Department KPIO Project sponsor Project manager Contractor
Costs Efficiency Quality Timeliness Funding Corruption Public accountability
Benefits of Results-Based Monitoring
Optimize project outcome performance Avoid mistakes from the past Priorities future project proposals Greater public accountability Can demonstrate good governance Have a performance culture Minimized project waste Set higher standards using Benchmarks
Results Monitoring Model
Develop industry
MaterialsEquipmentManpower
Construct roads New road
Industry grows
Road usage, Speedy, Smooth
Good transport access
Results Monitoring Indicators
New road
Industry grows
Road is used, Fast, Smooth
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
2000 2001 2002 2003
Vehicle users汽车使用者
Results Monitoring Indicators
New road
Industry grows
Road is used, Fast, Smooth
0
10
20
30
40
50
60
2000 2001 2002 2003
Averagetraveltime 平均旅行时间
What we want to know after construction ends and asset is being used
is the finished project being used as intended? is the operating performance as planned or do
we need to make some modifications to make it work better?
is the project achieving the intended results?
What we want to know later (>3 years after construction):
did the project achieve the intended operating results?
did the project achieve the intended impact? should we do similar projects in future? what lessons have we learned from this project
that we can apply to future projects?
Outcomes• Usage rate
– Average number of vehicles per day travelling on road.
– Output of productive unit:• e.g. MWh provided to industry
• Quality– Average travel time between A and B by road– quality of electricity output, e.g. no load shedding, xx
MW peak load maintained
• Financial– revenue from the asset usage (calculated as pay-
back period or rate of return FIRR)
Outcome indicator: Usage
Nanpu BridgeAverage number of vehicles per dayYear Forecast Actual
1992 19424 166121993 33026 199101994 34986 303001995 36848 455761996 35195 564561997 37272 627001998 39471 589001999 41799 600002000 44274 63500
Outcome indicator: UsageAverage number of vehicles per day
Nanpu Bridge
0
10,000
20,000
30,000
40,000
50,000
60,000
70,000
1992 1993 1994 1995 1996 1997 1998 1999 2000Year
Ve
hic
les
pe
r d
ay
Forecast
Actual
What does this tell us about the project?
Outcome indicator - Usage
What does this tell us about the project?
Indicator 1
0
50
100
150
200
250
300
350
1 2 3 4 5 6 7 8 9 10 11 12
Year after completion
Us
ag
e
Forecast
Actual
Outcome indicator: Operating Revenue
Financial PerformanceYear Forecast Actual
Toll revenues1992 - 16.3 1993 80.8 37.5 1994 84.7 77.3 1995 88.1 138.1 1996 82.9 170.6 1997 87.2 188.3 1998 91.0 175.1
Outcome indicator: Operating Revenue
Toll Revenues - Nanpu Bridge
-20.040.060.080.0
100.0120.0140.0160.0180.0200.0
1992 1993 1994 1995 1996 1997 1998
Year
Y m
illi
on
Forecast
Actual
What does this tell us about the project?
Outcome indicator: Operating profit
Financial PerformanceYear Forecast Actual Forecast Actual
Toll revenues Gross Operating Profit1992 - 16.3 - 9.9 1993 80.8 37.5 72.8 28.2 1994 84.7 77.3 75.7 62.0 1995 88.1 138.1 78.4 123.0 1996 82.9 170.6 72.4 152.0 1997 87.2 188.3 75.9 175.8 1998 91.0 175.1 78.8 163.8
Outcome indicator: Operating profit
Gross operating profit - Nanpu Bridge
-
20.0
40.0
60.0
80.0
100.0
120.0
140.0
160.0
180.0
200.0
1992 1993 1994 1995 1996 1997 1998
Year
Y m
illi
on
Forecast
Actual
What does this tell us about the project?
Case Studies – Objectives• Objectives:
1. Pilot Result-Based performance indicators, to bring international best practices into project monitoring and evaluation
2. Find practical ways of implementing them into the current computer system
3. Develop cases and materials for promotion of and training in Result-Based Monitoring & Evaluation
4. Provide learning experience with Result-Based performance indicators for KPIO, KPIO provincial level and executing agencies
Considerations for case study projects Sectors
Power Transport Hospital Others
Status Commenced project implementation Recently completed Completed 3 – 5 years
Qualities Good or OK Problematic
Desirable variety of case study projects
• Contract types– turn-key – Build-Own-Transfer– multiple contractors/consortium– etc…
• investment sources– Government– International loans– Private or foreign investment
• other– large/small– profit/non-profit
Steps
1. Choose case study projects.2. Gather documentation on project justification 3. Develop monitoring framework for each
project.4. Define results-based indicators for each
project.5. Collect data for result-based indicators for
each project.6. Assess indicators, change if necessary.7. Use case studies to develop training materials.
Case Studies
The project indicator monitoring framework The project indicator working sheet The main activities The results (Hebei Expressway) Some issues and findings in the case
studies.
Thank you! 谢谢!