Evaluation Office, December 2007United Nations Development Programme

ACHIEVINGRESULTS

RESULTS-BASED MANAGEMENT AT UNDP

EVALUATION OF

Copyright UNDP 2007, all rights reserved.Manufactured in the United States of America

Design: Suazion, Inc. (NY, suazion.com) Production: A.K. Office Supplies (NY)

Team Leader Derek Poate

International Consultants Paul BalogunMunhamo ChisvoRobert LaheyJohn Mayne

National Consultants Arcadie Barbaros ie (Moldova)Seheir Kansouh-Habib (Egypt)Farsidah Lubis (Indonesia)Kenneth Mwansa (Zambia)Oscar Yujnovsky (Argentina)

Evaluation Office Task Manager S. Nanthikesan

Advisory Panel Sulley Gariba, Former President, International Development Evaluation Association; Head, Institute for Policy Alternatives, Ghana

Peter Van der Knaap, Director of Policy Evaluation,Netherlands Court of Audit

Jehan Raheem, Professor, Brandeis University, Massachusetts,United States; former Founding Director Evaluation Office, UNDP

Odetta R. Ramsingh, Director-General, Public Service Commission,South Africa

Research Support Elizabeth K. Lang Nayma Qayum

Editor Margo Alderton

EVALUATION TEAM

EVALUATION OF RESULTS-BASED MANAGEMENT IN UNDP

F O R E W O R D i

This report presents the assessment of anindependent evaluation conducted by the UNDPEvaluation Office of the adoption of results-based management as an approach. Results-based management was introduced in UNDP in1999 and considerable experience has beengained by the organization since its introduction.The Executive Board, recognizing the need totake stock and build on this experience, approvedin its annual session in 2006 the inclusion of thisevaluation in the Evaluation Offices work plan.

The main purpose of this evaluation is toexamine the degree to which results-basedmanagement has fostered a results culture withinthe organization, enhanced capacity to makebetter management decisions, and strengthenedUNDPs contribution to development results.The intent of this study is not to assess whetherresults-based management systems are in place inUNDP, as compliance on management systems isreviewed by audit. This evaluation was carefulnot to duplicate the extensive studies and reviewsof results-based management in UNDP carriedout both by the organization and by partners.The study did not, therefore, focus on how results-based management systems are used in reportingon UNDPs performance to the Executive Boardor on the quality of the results frameworks usedor indicators selected. The core analysis of theevaluation is directed at how results-basedmanagement has been used to enhance UNDPscontribution to development effectiveness.

The scope of the evaluation extended to allgeographic regions, covering country, regional,global and corporate levels of programming andorganizational work. UNDPs experience in results-based management from 1999 to the present wascovered. Evidence for the assessment was drawnfrom case studies in five countries (Argentina,Egypt, Indonesia, Moldova and Zambia),interviews and focus-group discussions in UNDP

headquarters in New York, an electronic surveyof programme staff in country offices in allregions to which there were 365 responses, and adesk review of related evaluative literature.

UNDP was among the earliest UN organizationsto introduce results-based management and theevaluation finds that its experience has not divergedsignificantly from that of other public-sectoragencies. There are however, some specificchallenges that UNDP faces. The evaluationconcludes that UNDP is largely managing foroutputs rather than outcomes and that the linkagesbetween outputs and intended outcomes are notclearly articulated. The introduction of corporatesystems and tools, which have had some efficiencybenefits, have not, however, strengthened theculture of results in the organization or improvedprogrammatic focus at the country level. Thecurrent approach of defining and reporting againstcentrally defined outcomes tends to undermineUNDPs responsiveness and alignment to nation-ally defined outcomes and priorities.

The evaluation makes a number of recommenda-tions to address these and other challenges. Thereare important recommendations relating todeveloping a culture of results at all levels throughstrong leadership, changes in incentives, investmentin capacity, streamlining systems, and investing in and responding to performance audit andevaluation. Stronger oversight by the regionalbureaux is recommended, moving from the currentemphasis on process compliance to a substantiveengagement with country offices on the contentof country programmes and their contribution todevelopment results. The evaluation recommendsa shift to a lighter corporate results framework inthe Strategic Plan that provides clear boundariesregarding priority areas defined by the ExecutiveBoard while allowing and enabling countryoffices to define development outcomes withnational partners in line with national priorities

FOREWORD

F O R E W O R Di i

as identified in the United Nations DevelopmentAssistance Framework (UNDAF).

We are very grateful to the Executive Boardmembers, governments and civil societyrepresentatives in the case-study countries whowere very generous with their time and ideas.I would like to express our particular gratitude to all the resident representatives, UNDP staffand members of the UN country teams in thecountries visited by the team, as well as thecolleagues in New York who provided vitalfeedback to the team to enable them to reachtheir conclusions.

The report is the result of the dedication andintense team work of a number of people. TheEvaluation Office is deeply grateful to the team that prepared the report. The team leader,Derek Poate, developed the methodology for theevaluation and led the drafting of the report.Other members of the team were Paul Balogun,Munhamo Chisvo and Robert Lahey, each ofwhom participated in one or more case-studymissions and contributed to the evolution of the main report. The team benefited from theinvolvement of John Mayne who has extensivelystudied results-based management in the UNsystem. The case studies were strengthened bythe work of national experts including ArcadieBarbarosie (Moldova), Seheir Kansouh-Habib(Egypt), Farsidah Lubis (Indonesia), KennethMwansa (Zambia) and Oscar Yujnovsky (Argentina).

The Evaluation Office invited independent expertsto join an advisory panel for the evaluation. Themembers of the panel were Sulley Gariba (former

President of the International DevelopmentEvaluation Association and Head of the Institutefor Policy Alternatives, Ghana), Peter Van derKnaap (Director of Policy Evaluation, NetherlandsCourt of Audit), Jehan Raheem (former Directorof the UNDP Evaluation Office and Professor atBrandeis University, United States) and OdettaR. Ramsingh (Director-General of the PublicService Commission,South Africa). The final reportbenefited from the comments and suggestions ofthe advisory panel.

The evaluation was ably task managed bySuppiramaniam Nanthikesan, Evaluation Advisor,in our office. Other colleagues in the office madeimportant contributions to the report includingEvaluation Advisor Oscar Garcia, Michelle Sywho handled administrative support, and AnishPradhan who provided information technologysupport to the electronic survey and to thepublication process. Research support was providedby Elizabeth K. Lang and Nayma Qayum.

As the report points out, results-based managementis a journey not a destination and results-basedmanagement is just one piece in the complex setof efforts directed at achieving intended results. Ihope that this evaluation will be useful in helpingUNDP chart a course in this journey to managebetter for results that enhance developmenteffectiveness in the countries where UNDP works.

Saraswathi MenonDirector, Evaluation Office

C O N T E N T S i i i

Acronyms and Abbreviations v

Executive Summary vii

1. Introduction 1

1.1 Methodology 11.2 The evolving aid context 41.3 What entails results-based management in UNDP? 51.4 Structure of the report 8

2. Results-based Management in UNDP 9

2.1 The evolution of results-based management in UNDP 92.2 Intended effects on the organization 11

2.2.1 Setting strategic goals 112.2.2 Aligning results 122.2.3 Aligning funding 142.2.4 Aligning human resources 152.2.5 Monitoring for results 162.2.6 Adjustment and learning 172.2.7 Accountability 172.2.8 Guidance and capacity building for results-based management 18

3. Findings from the Evaluation 21

3.1 Results orientation 213.1.1 Evolution of strategic goals and effects on the organization 213.1.2 Alignment with strategic goals at the country programme level 22

3.2 Managing for results 273.2.1 Monitoring at the country programme level 273.2.2 Adjustment of resources 293.2.3 Relations with the regional bureaux and substantive corporate oversight 30

3.3 Evaluation, learning and accountability 333.3.1 Role and use of evaluation 333.3.2 Learning and technical support 343.3.3 Accountability 353.3.4 Definition of clear roles and responsibilities (accountability relationship) 353.3.5 Clear performance expectations and reward systems (transparent incentive mechanism) 363.3.6 Credible and timely measurement and reporting of the results achieved (giving account) 363.3.7 Mechanism to hold to account 363.3.8 Accountability for outcomes 37

4. Discussion 39

4.1 Implications of the wider environment 394.2 A culture of results is being developed 404.3 Factors helping and hindering a culture of results 404.4 Staff survey on results-based management 424.5 Early gains, lost momentum 424.6 Managing for outcomes was correct but unsupported 444.7 Results-based management is more than tools and systems 46

CONTENTS

C O N T E N T Si v

5. Conclusions and Recommendations 47

5.1 Recommendations 50

Annexes

1. Terms of reference 532. People consulted 593. Evaluation team and advisory panel members 664. References 675. Tables 716. Results-based management in development organizations 877. Sources of funding to UNDP 978. Summary of results from the staff survey on results-based management 1099. Results-based management benchmarks assessment 11110. Findings from the country case studies 123

Boxes, Tables and Figures

Box 1. Results-based management is a challenge faced by other development partners 6Box 2. Managing for results in associated funds and programmes 10Box 3. Broad, permissive outcome statement 24Box 4. Making outcomes evaluable 24Box 5. Monitoring and evaluation system components 28Box 6. Office of Audit and Performance Review 32

Table 1. Key events in the development of results-based management 9Table 2. Prospective cultural changes in UNDP 13Table 3. Percentage of programmatic funding by source 14Table 4. Key features in implementation versus outcome monitoring 16Table 5. Key strategy documents and evolution of UNDP goals 22Table 6. Core fund allocation strategy 25Table 7. Targets in the Moldova Strategic Notes (2004-2007) 31Table 8. Factors supporting and holding back a results focus 41Table 9. Themes associated with managing for results 44

Figure 1. Theory of change for results-based management 2Figure 2. Force field analysis of factors for and against a results focus in UNDP Zambia 41

A C R O N Y M S A N D A B B R E V I A T I O N S v

ATLAS UNDPs enterprise resource planning programme

BDP Bureau for Development Policy

CCA Common Country Assessment

CCF Country Cooperation Framework

CPAP Country Programme Action Plan

CPD Country Programme Document

M&E Monitoring and Evaluation

MDG Millennium Development Goal

MfDR Managing for Development Results

MYFF Multi-year Funding Framework

NHDR National Human Development Report

OECD-DAC Organisation for Economic Co-operation and Development-Development Assistance Committee

OSG Operations Support Group

PRINCE 2 Projects in Controlled Environments (a process-based method for effective project management)

RBA Regional Bureau for Africa

RBAP Regional Bureau for Asia and the Pacific

RBEC Regional Bureau for Europe and CIS

RBLAC Regional Bureau for Latin America and the Caribbean

RCA Results and Competency Assessment

ROAR Results-oriented Annual Report

RSC Regional Support Centre

SURF Sub-regional Facility

UNCT UN Country Team

UNCDF UN Capital Development Fund

UNDAF UN Development Assistance Framework

UNFPA UN Population Fund

UNIFEM UN Development Fund for Women

UNV UN Volunteers

VDA Virtual Development Academy

ACRONYMS AND ABBREVIATIONS

E X E C U T I V E S U M M A R Y v i i

INTRODUCTION

This report sets out the findings of an evaluation ofthe adoption and use of results-based management.The evaluation has focused on the organizationalstrategy, vision and expectations of the results-basedmanagement approach; the design, implementa-tion and use of the system to operationalize thisapproach; as well as the results of this effort. Thescope of the study covers the period 1999-2006,all geographic regions, and the adoption of results-based management at the programme, country,regional and corporate levels.

An inception report prepared by the evaluationteam proposed a theory-based methodology. Thetheory identifies a causal process with five keyelements. This five-stage process provides thestructure for enquiries:

n Setting a strategic framework to describe thedesired results

n Developing programmes aligned to thestrategic results framework

n Measurement and analysis of the actualresults achieved

n Use of information to improve design anddelivery of programmes

n Reporting on performance as part of theaccountability process

Information was gathered through interviewswith staff in UNDP and other United Nationsorganizations at headquarters; review of UNDPdocumentation and that of other organizations;visits to five country programmes (in Argentina,Egypt, Indonesia, Moldova and Zambia), whereinterviews were held with a wide range ofstakeholders in the United Nations system,government, civil society and other developmentpartners; and through an electronic survey ofUNDP staff in country offices. The choice ofcountries did not include conflict-affected

situations, a specification in the original terms ofreference, and this limits the ability to generalize,in the conclusions and recommendations, aboutthe use and impact of results-based managementin those contexts. A full list of people consultedin all locations is presented in the report.

THE EVOLVING DEVELOPMENT CONTEXT

The 1990s marked a shift from aid flows beingdetermined by geopolitical considerations tohaving a focus on the promotion of sustainablehuman development. That shift was accompaniedby a steady decline in official developmentassistance and increasing pressure to demonstratethe effectiveness of aid. UNDP adopted results-based management with the clear intent of reversingthe declining resource base, assuring predictabilityof programme funding and demonstrating aperformance focus to donors.

Since then, the role of UNDP has changed frommainly funding and implementing downstreamactivities to emphasizing upstream activitiesinvolving advocacy, policy support and capacitystrengthening, using national execution as thepredominant mode of delivering assistance. Thishas been reinforced by the changing environmentfor development cooperation. The increasingneed for country-based joint assistance strategies,as was emphasized by the Paris Declaration, hasled to new aid modalities, such as direct budgetsupport and sector-wide approaches.The decline inresources has been overcome, and programmaticservices delivered by UNDP increased from $2 billion in 1999 to $4.36 billion in 2005.

INTRODUCTION OF RESULTS-BASED MANAGEMENT

For results-based management to be successful,organizations need to develop and nurture aculture of results where enquiry, evidence and

EXECUTIVE SUMMARY

E X E C U T I V E S U M M A R Yv i i i

learning are valued as essential to good manage-ment. The use of results information in managingand informing decision-making is usually seen as the main aim of introducing results-basedmanagement. Managers are expected to: under-stand why projects and other activities contributeto the outcomes soughtthe theory of change, setmeaningful performance expectations, measureand analyze results, learn from this evidence toadjust delivery and modify or confirm programmedesign, and report on the performance achievedagainst expectations. When results-based manage-ment was introduced in UNDP, it was seen asinvolving all the features previously listed, andthe importance of building a culture of resultswas recognized.

In many respects, results-based management wasa logical continuation of management reformsfrom the 1990s. A large number of tools andsystems within UNDP are now associated withresults-based management, but these emergedwithout a guiding design over a 10-year period ofinnovation, redesign and change, which has attimes been unsettling for country office staff.

Results-based management is not without itscritics. In trying to set clear, concrete objectivesand targets, political scientists argue that results-based management can conflict with the need tokeep objectives sufficiently imprecise to gainwidespread support. Another criticism is thatmany of the developmental results sought byUNDP cannot easily be measured. As a result,results-based management forces measurementand reporting of less important results, especiallyoutputs. When an organization overemphasizesany set of performance indicators and targets,the staff tend to become preoccupied with thoseindicators and targets rather than the wider results.

It is important to take these concerns intoaccount when developing and, especially,managing the results-based management regime.

GOALS AND PROGRAMME FOCUS

One of the most visible elements of the results-based management approach was the adoption of

multi-year funding frameworks (MYFFs), whichhave strategic goals designed to help focus theprogramme and improve communication withexternal stakeholders. Alignment of countryprogrammes with strategic goals was furtherpromoted by a shift in focus from project outputsto outcomes. In a parallel initiative, a series ofAdministrators business plans introduced plansto change the culture of the organization and putforward the balanced scorecard as a tool toreport against a broad range of physical andfinancial indicators of operational change.

This evaluation has found that the goals ofUNDP in the strategic frameworks have changedin presentation, but the underlying areas of work have remained almost the same as before.The focus areas under the goals have been ration-alized and simplified, but it is hard to identifysubstantive change to the scope of activities at thecountry level. Managers and staff in countryoffices believe that the MYFFs have helped tobring focus and improve positioning and advocacy.They have been a positive tool in conjunctionwith the reformed United Nations DevelopmentAssistance Framework (UNDAF) and countryprogramme document to foster dialogue aboutcountry results. However, their effect on countryportfolios has been limited to encouraging theremoval of outlier activities. Projects have justbeen mapped to the new frameworks.

The reasons for these limited effects are several.The MYFF sub-goals and service lines were verygeneral and were not implemented with strictlymonitored hard boundary rules about whatcould and could not be justified under them.Intended outcomes for country programmes alsotended to be vague, rather than rigorous statementsforcing consideration of what projects would bestcontribute to their achievement. In other words,outcomes were defined to cover the existingportfolio rather than drive decisions as to howthe portfolio should be refined and developed.

Results-based management assumes that managershave the flexibility to allocate programmeresources to maximize results. That assumptiondoes not generally hold true for UNDP. Core

E X E C U T I V E S U M M A R Y i x

programme resources, the one area where managershave some financial flexibility, may have increasedduring the evaluation period, but core programmeresources have become a progressively smallerpercentage of total programme funds. Countryoffices have thus become more dependent onnon-core programme financing, which isrelatively inflexible, and often find that the coreprogramme resources are only sufficient to act asseed money.

Flexibility to change country office organiza-tional structures has the potential to foster closerlinkages to results. Development and investmentin thematic practice areas was intended to fosterskills. Twenty-five offices have appointedmonitoring and evaluation (M&E) specialistsand 10 have an M&E unit. There is evidence thatcountry offices organize their staff for delivery ofresults, with many structuring programme staffteams around broad programme areas. TheResults and Competency Assessment for staff hasthe potential to foster a results and accountabilityculture. But in practice, mobilization of resourcesand delivery are more powerful drivers of individualperformance among programme staff thanachievement of outcome level results. In its currentmodality, the Results and Competency Assessmentdoes not support managing for results.

MONITORING AND ADJUSTMENT

The results-oriented annual report wasintroduced to monitor country programmes, butit has become primarily a tool for reporting tosenior managementof little use for countryoffice or regional managementand is viewed asthe data source for the annual MYFF report tothe Executive Board. That report, in turn, iscriticized by Board members as being too vagueand containing little substantive detail on results.With the advent of results-based management,the focus of results shifted to outcomes, but apartfrom the results-oriented annual report, nospecific tools were developed to help monitorresults. The ATLAS system is steadily gaining inimportance, but its primary focus is financialmanagement and project monitoring.

The removal, in 2002, of many mandatoryproject monitoring tools and procedures wasconsidered a risk, but it was done in the hope thatcountry offices would continue with projectmonitoring as well as moving into outcome-levelmonitoring. The evidence suggests that the movemay have led to a decline in project-level M&Ecapacity in some country offices, but it hasstimulated the creation of diverse M&Eapproaches in othersespecially where there is astaff member dedicated to M&E. While someprogress has been made in country officestowards monitoring outcomes, approaches fail to explain how projects are contributing toprogramme outcomes.

Evidence from countries visited also shows thatcountry offices face challenges in definingoutcomes that are owned by other partners,although the UNDAF/country programmedocument/country programme action planprocess is a help, as are joint assistance strategies.This reflects the reality that many partnergovernments and donors have their own M&Esystems, which may or may not use results-basedmanagement. The clearest indicator of this is theabsence of joint outcome evaluations and thelimited use by UNDP of M&E evidence derivedfrom other systems of partners.

The oversight and management roles of theregional bureaux do not focus on results. Formalpoints of interaction with country offices areconcentrated in the development of new countryprogrammes, but the interaction focuses onprocess conformity rather than on substance.Thereafter, the engagement is generally driven byresource mobilization and delivery. Routineinteraction between country offices and therespective bureaux takes place through deskofficers in the regional bureaux, who are animportant link in understanding countryperformance but are overworked, junior in status,and often lack resources and technical skills.

In order to address shortcomings in the results-oriented annual report, the regional bureaux have approached information management indifferent ways, often drawing on the new

E X E C U T I V E S U M M A R Yx

information tools. But the quantitative informa-tion in ATLAS and other tools is not accompaniedby qualitative information that explains perform-ance. Some regional bureaux argue that the shiftin focus to outcomes has not been matched byM&E tools to measure performance. The currentsystem is dominated by self-reporting throughthe results-oriented annual report and theResults and Competency Assessment, withoutadequate mechanisms for periodic validations ofquality.

Overall, the evaluation finds that there is littleevidence to indicate that results are being usedsystematically to inform adjustments to thecountry portfolios.

EVALUATION, LEARNING AND ACCOUNTABILITY

The adoption of results-based management in1999 was followed, in 2002, by the introductionof outcome-level evaluations. Adoption ofoutcome-level evaluations in the countries visitedhas been slow. In general, they have been under-budgeted and poorly timed to influence thecontent of country programme documents.Independent evaluation is an important elementof results-based management to validate perform-ance. Yet country evaluations conducted by theUNDP Evaluation Office have emphasizedlearning over accountability and have notmeasured performance against stated intentions.Validation occurs only through infrequent audits.However, the role of evaluation was strengthenedin 2006, when the Executive Board endorsed anew evaluation policy.

Little use is made of results for the purpose oflearning at the country level, and staff would liketo see more time allocated to this process. AtUNDP headquarters, the Bureau for DevelopmentPolicys (BDP) practice architecture is used fortechnical support, but this architecture has poorlinkages to build on lessons emerging fromevaluations and has produced few productsclearly tailored to business processes.

The UNDP accountability framework does not

support results-based management. Roles andresponsibilities are generally clear, but countryprogramme outcomes and indicators are notsubject to quality assurance and there is littleindependent validation. Individual targets in theResults and Competency Assessment are self-selected and are often applied retrospectively andpoorly linked to incentives. Despite the intendedshift to managing for outcomes, individual staffremain tied to a project orientation and account-ability for outputs.

There is no evidence that the Resident Repre-sentative/Country Director is held accountable formanaging for outcomes, and there is considerablescepticism within UNDP over whether this isfeasible, despite evidence of moves towards such anapproach in sister organizations such as UNFPA.

MANAGING FOR RESULTS IN THE WIDER CONTEXT

UNDP works in a multilateral context, in whichits mandate emphasizes the centrality of nationalownership and the role of UNDP in buildingnational capacity. This implies the need to workthrough national systems, where feasible. UNDPprogrammes invest significant attention insupporting national systems for tracking progresstowards the Millennium Development Goals andother development results. But there is littleevidence of engagement with national planningand results/performance systems at the sector orthe programme level.

This suggests two important things. First,UNDP has missed opportunities to harmonizewith the results-based management approachesof national partner governments. This reducesthe scope for national capacity development andfor enhancing national ownership. Second,UNDP has not fully considered the implicationsof its results-based management approach onbroader United Nations reform initiatives such asthe Paris Declaration and the MonterreyConsensus, which emphasize closer alignmentand harmonization with governments anddevelopment partners.

E X E C U T I V E S U M M A R Y x i

The shift in strategic direction and the newresults framework brought early gains to theprogramme through a clearer expression ofUNDP roles and functions. Staff have foundstrategic frameworks to be valuable in discussionswith governments and development partners.But the goals in the results frameworks for thetwo MYFFs and the draft strategic plan are toobroad to help UNDP focus support in areaswhere it has a comparative advantage. Thestrategic objectives have not been used to definewhat is and what is not appropriate for countryprogrammes to support.

Sharpening the focus in this regard would requirea change in the relationship between regionalbureaux and country offices. The focus wouldhave to shift away from oversight of processesand resources, to a greater emphasis on substan-tive content. The low level of core funding andhigh reliance on non-core funds by many countryprogrammes means that the managementspotlight needs to be on the degree to whichResident Representatives manage to refocus inthis situation.

In the inception report for this evaluation, a tableof benchmarks based on international performancestandards for results-based management wasintroduced as an evaluation tool. The reportassesses each benchmark, drawing on thefindings of the present study and relevantindependent studies. Unfortunately, in theirpresent form, it is not possible to use thebenchmarks to assess the status of UNDPrelative to other United Nations organizations orto the wider population of public-sector organi-zations that use results-based managementapproaches. But progress has been made in mostareas. Of the 21 categories assessed, two areassessed as fully achieved, 16 as partiallyachieved, and three as not achieved. The largenumber of benchmarks assessed as partiallyachieved reflects the positive work of UNDPtowards creating an architecture to manage forresults, but it also reflects that too many elementsof the approach are not functioning effectively.

CONCLUSIONS

Overall, UNDP has established a cycle of settingand revising corporate goals, introduced improvedoffice systems to manage project finances,institutionalized the need to report on corporateand individual performance, and raised awarenessabout results throughout the organization.

Conclusion 1: The experience of UNDP withintroducing results-based management issimilar to that of other organizations.

UNDP was one of the first United Nationsorganizations to move to a results-based manage-ment approach. Review of the literature stronglysuggests that the experience of UNDP did notdiverge significantly from that of many otherpublic-sector organizations.The present evaluationidentified a number of areas where greaterprogress could have been made, but even underperfect conditions, it is unlikely that UNDPcould have fully institutionalized a results-basedmanagement approach within eight years.Subsequent conclusions and the recommendationstherefore focus on the key challenges for UNDP.

Conclusion 2: UNDP has a weak culture of results.

Adopting results-based management was alogical continuation of the management reformsthat occurred in the 1990s and a response topressure to improve performance across theUnited Nations as a whole. Significant progresshas been made on a number of fronts, sensitizingstaff to results and creating the tools to enable afast and efficient flow of information. Managingfor results has proved harder to achieve. In partic-ular, the strong emphasis on resource mobilizationand delivery; a culture that does not support risk-taking; systems that do not provide adequateinformation at the country programme level; alack of clear lines of accountability; and a lack ofa staff incentive structure all work againstbuilding a strong culture of results.

E X E C U T I V E S U M M A R Yx i i

Conclusion 3: The corporatist approach hashad only a limited effect on developmenteffectiveness at the country level.

UNDP adopted a systems approach to stimulatemanaging for results, which meant that changewas to be driven by the implementation ofcentrally designed and prescribed systems. Thesewere developed primarily to enable aggregatereporting of UNDP performance to theExecutive Board while at the same time creatinga clearer focus for the programme.

In practice, the corporate goals and service linesset by headquarters have proved too numerous,with very permissive definitions. This has led tocountry offices manipulating their programmesto fit into corporate service lines, divertedattention away from country needs and madereporting to the Executive Board more aboutprocess than substance. There is little evidencethat this approach has significantly affected theshape of country-level programmes, but there issignificant evidence that it has imposed unnecessarytransaction costs at the country level. A notableomission is the lack of oversight systems thatfocus on tracking whether programmes use resultsto adjust resources such as people, money andpartnerships in order to improve future results.

Conclusion 4: Results-based management hasbeen misinterpreted as not supporting thedecentralized way in which UNDP works.

UNDP works in a strongly decentralized way,yet the results frameworks in the MYFF were not geared to country processes. Emerging new systems following the reform of countryprogrammes, including the UNDAF, the country programme document and the countryprogramme action plan have the potential tocreate objectives for United Nations organiza-tions that are aligned with national plans andresponsive to country needs.

Decentralization has been accompanied bydelegation of authority over the countryprogramme. Under current procedures, country

programmes are not scrutinized for developmentpotential by regional management, an abdicationof responsibility. As a result, evaluation andauditing are the only means to check that countryprogrammes are contributing to corporate goals.

The top-down approach has inadvertentlyfuelled concerns that having corporate goals is ameans of imposing upon programmes at thecountry level. The role of results-based manage-ment is not to constrain the ways in whichprogrammes are negotiated at the country levelbut to provide a framework, so that UNDP workswithin its mandate and ensures that resources arealigned with achieving results. Once programmesare agreed upon at the country level, results-based management should provide standards fordialogue about how to set realistic outcomes,select objective indicators that demonstrateprogress towards development objectives, andjointly monitor progress.

Conclusion 5: Results-based management systemsare not helping build a results-based culture.

There are strong perceptions within UNDP thatfinancial administration and management systemshave improved. However, there is little evidencethat these systems have led to an increased focus onmanaging for outcomes. ATLAS and PRINCE2both deal with information at the project level,and the project is at the core of their designs. TheResults and Competency Assessment does noteffectively incorporate key results that reflectsuccessful management for results by individuals.There are also concerns that systems havebecome too complex and time-consuming.

Results systems have been designed mainly tomeet the demand for data for reporting to theExecutive Board rather than to manageoutcomes. Yet UNDP has not developed a systemfor reporting on its contribution towardsdevelopment results. This reflects a number ofissues. The corporate-level results frameworkshave never included high-level goals withsubstantive measurable and agreed-upon indica-tors against which to assess global progress. The

E X E C U T I V E S U M M A R Y x i i i

goal-level reporting by UNDP contrasts starklywith the objectivity of reporting against theMillennium Development Goals. UNDP hasdeveloped a reporting system that aggregateswhether or not results will be delivered whenexpected. This approach has limitations: thecountry programme outcomes against whichUNDP is expected to deliver are poorly defined;the logic linking outputs delivered by UNDPwith achievement of the outcomes is often notexplained; and therefore, this reporting systemfails to report on UNDP performance relative towhat it is accountable for.

Conclusion 6: Managing for results requiresleadership.

The importance of leadership to drive results-based management forward has been noted inseveral parts of the present report. A goodexample of effective leadership was the role of theprevious Administrator in fighting decliningresources. A strong personal commitment wassupported by: a single, simple and consistentmessage on resource mobilization that wascommunicated to both internal and externalaudiences; development of systems to track,measure and report managers success in mobilizingresources; and a clear perceived link betweensuccessful resource mobilization and advance-ment within the organization.

The same drive and visible, consistent senior-levelsupport is needed for results-based management.Four relationships stand out as the most critical:at the Executive Board, to ensure the programmeis held to account for development results;between the Administrator or AssociateAdministrator and the Bureau Directors; betweenDirectors of Regional Bureaux and ResidentRepresentatives or Country Directors; andbetween Resident Representatives or CountryDirectors and staff within country offices.

RECOMMENDATIONS

Managing for results is a dynamic process, andmany of the issues raised in this report are known

to UNDP management and are receivingattention. There is genuine interest and supportat the country level for a better focus on results.

Recommendation 1: Strengthen leadership and direction.

The first and overarching recommendationaddresses the need to capitalize on what has beenachieved to date and establish a stronger cultureof results. The success of this is not dependentupon tools and systems, but leadership and direction.Sustained commitment by top management, theAdministrator and the Associate Administratoris required.

Strong leadership is necessary. Attention to thefollowing issues is also needed: a shift in theaccountability framework from process andcompliance to results; outspoken commitment bysenior management, especially the Directors of Regional Bureaux; a change in dialoguethroughout the organization that prioritizesmanagement for development results andaddresses how this will be balanced againstcompeting demands such as resource mobilization;time and space for staff to give feedback on andlearn from experiences; a shift in organizationalpractices to take risks and manage for outcomesrather than outputs; and improved capacity tomeasure results.

Recommendation 2: Global goals, localsolutionsSharpen the role of the strategicresults framework.

Management should adopt a results frameworkthat distinguishes more clearly between corporategoals and country programme outcomes.

For the four UNDP focus areas, objectives shouldbe based on the key results areas, with indicatorsof substantive development change comparableto those used for the Millennium DevelopmentGoals. The corporate key results areas contain thebasis of what could be measurable goal-levelobjectives, for example: promoting inclusivegrowth; promoting gender equality; fostering

E X E C U T I V E S U M M A R Yx i v

inclusive participation (in governance); andempowering the poor, women and youth. Thisapproach will take time. The Executive Boardand UNDP should start with those key resultsareas where internationally agreed-upon indica-tors already exist.

Identifying and reporting on UNDP contribu-tions should not be an obstacle, any more than itis for organizations reporting country progressagainst the Millennium Development Goals.The development of robust models that show thelinks between country programme outcomes andUNDP contributions with achievement of thesehigh-level objectives is key.

The current practice of setting corporateoutcome-level objectives and indicators withinthe strategic plan should end. Instead, outcomeobjectives and indicators should be set at thecountry programme level, where they should belinked to UNDAF outcome objectives in thecontext of agreed-upon national developmentobjectives. Comparable outcome objectivesshould be set within the regional and globalprogrammes.

The above change would reinforce the decentral-ized nature of UNDP activities and build onUnited Nations reforms. The change would haveto be supported by a shift in the oversight roles of the regional bureaux, senior management andthe Executive Board away from compliance with procedures towards ensuring that countryprogrammes implement robust, results-basedmanagement approaches and are designed tocontribute to the UNDP focus areas.

Recommendation 3: Support managing foroutcomes at country offices.

Managing for outcomes means that managerslearn from results and empirical evidence and usethat evidence to adjust either the projects undertheir control or the composition of the portfolioof projects to maximize the contribution of theorganization to that outcome.

Implementing such an approach requires thatUNDP consider the wider environment at thecountry level when defining outcomes. There is aneed for improved guidance on how to balancedemands on the results-based managementsystem to meet internal UNDP needs with thoseimposed by the wider environment within whichUNDP operates at the programmatic level. Thisincludes dealing with three core issues raised inthis report:

n Ownership of results at the country level

n The implications of harmonizing otherpartners results-based management approachesand systems

n UNDP accountability for managing for results

The positive effects of some of the newlydeveloped UNDP systems are noted above, withthe caution that they are based predominantly on managing projects. Introduction of newmanagement and reporting systems will imposesignificant costs on country programme teams,and the country-level perception is that there hasbeen insufficient appreciation at the corporatelevel of the impact of these costs.

Country offices want to be effective and needsupport in several ways:

n A streamlining of systems, aiming for a moreuser-friendly integrated approach with betterprioritization and introduction of newrequirements across the organization

n Improved practical tools and guidelines toplan how projects will contribute toprogramme outcomes and to improve thespecification of indicators

n A large-scale capacity-development programmeto improve staff knowledge and skills

n Improved design of programmes based onproven models of intervention that can betailored to country circumstances, managed,monitored and evaluated

n Introduction of quality assurance to examinecountry programmes and assess evaluability

E X E C U T I V E S U M M A R Y x v

n Expanded use of country office outcomeevaluation plans geared to joint evaluationswith government and development partners

n Revision of the results-oriented annualreport to improve the evidence-base andstructure of the report

Recommendation 4: Expand investment anduse of evaluation and performance audit.

Improving country programmes requiresattention to detail and development of soundobjectives and indicators. A quality assuranceprocess is recommended as an ex ante way ofscrutinizing country programmes. This needs tobe supported by independent review of processesand compliance, along the lines of the currentenhanced audits.

The structure of results proposed here placesmore responsibility on country offices to develop

programmes that respond to country needs andcontribute towards global goals. It also frees themfrom having to fit into centrally determinedservice lines. The test, therefore, is whether theprogrammes that are developed contribute to thegoals of UNDP. This will require a strongerevaluation function that addresses both learningand accountability. The 2006 evaluation policy isa step in the right direction. The challenge now isimplementation that supports accountability andthe new results management guidance.

The above recommendations are intended to bemutually reinforcing and ought to be viewed as awhole. Some recommendations focus on overallframework rather than specific tools or issues.Dealing with leadership, the results framework,programme focus and accountability of theregional bureaux are the highest priority, followedby tools to help country offices chart contribu-tions to outcomes, and quality assurance systemsfor programme review.

C H A P T E R 1 . I N T R O D U C T I O N 1

This report sets out the findings of an evaluationof the adoption and use of results-based manage-ment. UNDP adopted results-based managementin 1999. There is considerable organizationalinterest in reviewing UNDPs experience asevidenced by the number of reviews beingconducted on related issues.1 In addition, recentevaluations conducted by the UNDP EvaluationOffice have highlighted persistent issues andgaps related to the operation of results-basedmanagement systems and approaches at thecountry office level.

The Concept Note for the evaluation states thatthe evaluation will focus on the organizationalstrategy, vision and expectations of the results-based management approach; the design, imple-mentation and use of the system to operationalizethis approach; as well as the results of this manage-ment approach.2 In doing so, the evaluation aimsto provide feedback on UNDPs efforts to strengthenthe existing results-based management practicesand make forward-looking recommendations.The evaluation will not examine the tools ofresults-based management in any detail as thesehave been reviewed in other studies.

The scope of the study is quite broad. It coversthe period 1999 to 2006, all geographic regions,and the adoption of results-based management atthe programme, country, regional and corporatelevels. In the choice of countries that were visited

for this evaluation,3 this study has assessed theresults-based management approach under diversedevelopment conditions in which UNDPfunctionsincluding countries with very highaid dependence and varying capacities for M&E.This review also takes note of any specificcharacteristics peculiar to the specialized fundsfor UN Volunteers (UNV), UN DevelopmentFund for Women (UNIFEM) and the UNCapital Development Fund (UNCDF).4

1.1 METHODOLOGY

An Inception Report prepared by the evaluationteam proposed a theory-based methodology anddata collection using three research methodologies:5

n A desk review of relevant secondary material

n Five case studies of countries and theirregional bureaux

n Interviews with staff in headquarters and asurvey of staff in all country offices to gatherdata on issues emerging from the desk reviewand pilot-country case study

The theory of change for results-based manage-ment was derived from literature on theintroduction of results-based managementsystems and is presented diagrammatically inFigure 1.6 It identifies a causal process with fivekey elements. This five-stage process provides thestructure for enquiries.

Chapter 1

INTRODUCTION

1 MSI, Background Paper on Results-Based Management in Development Organizations, Management SystemsInternational, July 2006, p 6; Danish Ministry of Foreign Affairs, Assessing Results Management at UNDP, 15 June2006; UNDP Administrators Office, Management and Workflow ReviewPhase I, January 2006.

2 UNDP, Concept Note: Evaluation of Result Based Management at UNDP, UNDP Evaluation Office, New York, NY,February 2007. See also the Terms of reference in Annex 1.

3 Argentina, Egypt (pilot country), Indonesia, Moldova, and Zambia.4 All three were added to the scope during the pilot country visit.5 Poate D, Balogun P, Evaluation of Result Based Management at UNDP, Draft Inception Report, March 2007.6 RBM in UNDP: Overview and General Principles, not dated.

C H A P T E R 1 . I N T R O D U C T I O N2

1. Set out a strategic framework that describesthe objectives and desired results of theorganization and the strategies to be used toachieve those results.

2. Develop programmes and sub-programmesin the organization aligned to the strategicresults framework, showing more specifics onthe results expectedresources, outputs, and

Figure 1. Theory of change for results-based management

Programmeorientation

Primary institutionaleffects

Secondaryinstitutionaleffects

Goal

Managing for Results

Improved Performance:organizational & development

effectiveness

Institutionalincentives

Externalfactors

and risks

Externalfactors

and risks

Organizationallearning

Set strategic goals for UNDP

Goals create a framework to defineprogrammes, organizational structure

and management arrangements

Plan specific results to achieve the strategic goals

Align resources (people, money and partnerships) to achieve results

Monitor implementation for progress and performance

Timely informationgoes to

decision-makers

Knowledgeable interpretation

Efficient adjustment of resources (people, money and partnerships)

to achieve results

Evaluate strategicgoals

Decision-makerresponds

C H A P T E R 1 . I N T R O D U C T I O N 3

the logic, sequence and timing of outcomesexpected to lead to the accomplishment ofthe programme objectivesand how theresults are to be measured.

3. Measure and analyze results achieved and thecontribution being made by the programmeto the expected results through both ongoingmonitoring and periodic evaluations.

4. Use the results information gathered to improvethe design and delivery of programmes.

5. Report on the levels of performance achievedas part of the accountability process.

The diagram follows the key principles forresults-based management at UNDP. The stagesin the diagram identify key effects, starting witha clearer orientation of the programme as awhole, followed by realignment of resourcestowards results, efficient adjustment of resourcesand links with knowledge and institutionallearning. An over-arching condition is having anorganizational climate or culture that encouragesmanaging for results. This issue is developedfurther with practical examples in Annex 6.

The implicit goal is improved performance(interpreted here as organizational effectivenessand contribution to development effectiveness).Details of the pathway by which results-basedmanagement processes that improve managementdecision making and stimulate developmenteffectiveness are not clearly specified in theliterature. This is methodologically challengingcompared with the situation in many otheragencies, since UNDP programmes are usuallyimplemented by development partners who, inmanagement terms, are at arms length fromUNDP and UNDPs contributions are oftensmall in scale, and work through policy adviceand advocacy, as well as money.

Evaluating results-based management systemsand processes within UNDP poses two signifi-cant challenges:

n At present, there are no internationally agreed-upon standards that define what should be

included within a results-based managementsystem and how such components shouldoperate in practice.

n Initial discussions with headquarters stake-holders and review of discussion on thePractice Networks indicate that there is not aconsensus within UNDP on what theresults-based management system is, how itshould operate and the intended effects.

The evaluation team therefore defined a set ofbenchmarks, drawing on three sources ofperformance expectations, to assess whethersystems are in place and the expected processesare being used in their operation. First, theevaluation team adopted the benchmarksdeveloped during the UN Joint Inspection Unitsreview of results-based management within theUnited Nations in 2004. These benchmarksprimarily focus on the existence and operation ofrelevant systems. Second, the team developedbenchmarks building on material presented inthe Joint Venture on Managing for DevelopmentResults Source Book. These benchmarks, whichare less precise in nature, mostly focus on assessingwhether systems are applied using a soundapproach. Third, the team developed additionalbenchmarks using the objectives set out in theParis Declaration on Aid Effectiveness, dealingprimarily with issues such as use of countrysystems, alignment and harmonization. Theseprovide a more forward-looking orientation to helpground the evaluation in the evolving contextwithin which UNDP operates. The benchmarksare set out in Annex 9 together with findingsfrom the evaluation, as discussed in Chapter 5.

Extensive consideration was given to theselection of case-study countries. A purposiveapproach was adopted, dictated by looking forcases that would provide evidence to test againstthe theory, rather than using a randomizedsampling approach. Specific criteria that wereassessed include:

n Countries where key informants believe thatthe office has used results-based managementapproaches to some extent. All managers will

C H A P T E R 1 . I N T R O D U C T I O N4

use information to make decisions, but thefocus of the evaluation is on the extent towhich information from the results-basedmanagement systems is used.

n Size of the UNDP programmepreviousanalyses and interviews with UNDPHeadquarter staff confirm that the size of theprogramme dictates whether or notsomebody will be dedicated full time withinthe UNDP country office to ensure theimplementation and smooth functioning ofthe results-based management systems.

n A broad range of development context (suchas high aid dependence and level of income),and taking account of audit reports on theprocesses followed.

n Representation across the regional bureaux,on the assumption that the managementstyles across the regional bureaux vary andthis may have affected how results-basedmanagement has been institutionalized.

There was neither time nor enough informationto select case studies that could be considered arepresentative sample of the entire population.7

Therefore, following case-study theory, in orderto generalize from the findings of the case studiesand avoid bias, the evaluation used the evaluationfindings to generalize from the theory underpin-ning results-based management and as outlinedin the Theory of Change. The unit of analysis forthe case studies was the country programme andthe relevant regional bureau. The final countriesselected were: Argentina, Egypt, Indonesia,Moldova and Zambia. This selection allowed fora better understanding of how decision making isaffected by results-based management across thechain of accountability within the organization.However, the evaluation did not examine results-based management in conflict-affected situations,a specification in the original terms of reference,which limits the ability to generalize in the

conclusions and recommendations about the use and impact of results-based management insuch contexts.

The primary data collection methodologies usedwere individual interviews with stakeholdersboth within and outside of UNDP and the use of group exercises to elicit country office views in key issues. Annex 2 is a list of all peopleinterviewed. This was supplemented by ananalysis at headquarters of how results-basedmanagement systems were developed, how that development was coordinated with thedevelopment of other key systems (mainly thehuman resources, knowledge management andfinancial allocation and monitoring systems), andhow their implementation was supported acrossthe organization.

Additional data was collected via an electronicsurvey of all management and programming staffin country offices (excluding the five countriesvisited). The survey was similar in style to theregular global staff surveyasking respondents ifthey agree or disagree with a series of statementsabout how results-based management functionsand the culture of results. A full list of the questionsappears in Annex 8 together with a summary ofresults. The questionnaire was sent to approxi-mately 1,700 staff; 365 replied.8

1.2 THE EVOLVING AID CONTEXT

In recent years within the international developmentcommunity, there has been enhanced cooperation toreduce poverty and work to increase developmenteffectiveness. The MDGs have set new standardsfor multilateral organizations, donors and partnercountries. For at least a decade, the internationalcommunity has been developing partnershipapproaches to development assistance, such assector approaches and poverty reduction strategies.Some of these approaches, such as general budget

7 Yin RK, Case Study Research: Design and Methods, 3rd Edition, Applied Social Research Methods Series, Volume 5,Sage Publications, 2003.

8 The anonymity of the survey system means that details of who did or did not reply are not known.

C H A P T E R 1 . I N T R O D U C T I O N 5

support, have been the subject of intense debateregarding their effectiveness. Aid effectivenesshas also been a subject of increased attention overthe last few years, with agreements to worktowards better harmonization, alignment andresults. At the same time, policy makers havetried to reach beyond development assistance toconsider the impact of other policy measures andprivate-sector activities, especially the impact offoreign direct investment and remittances.

The 1990s saw a shift in the thinking andpractice of development cooperation from aidflows being determined by national strategicconsiderations to a focus on the promotion ofsustainable human development.9 This shift also resulted in a steady decline in officialdevelopment assistance and increasing pressurefrom the public in donor countries todemonstrate effectiveness of aid. In response,some bilateral organizations (led by USAID),and the public sector in some donor countries(such as Australia, Canada and New Zealand)began to adopt a results-based managementapproach that was widely used in the privatesector. To reverse the declining resource base,assure predictability of programme funding anddemonstrate performance focus to the donors,UNDP also adopted results-based management.

Since adopting this new management approach,UNDP has had to deal with both quantitativeand qualitative changes affecting the aid environ-ment, including:

n Expanded operational capacityProgrammaticservices delivered around the globe byUNDP increased from USD 2 billion in1999 to USD 4.36 billion in 2005.

n The changing role of UNDPThe organiza-tion has shifted from mainly funding andimplementing downstream activities toemphasizing upstream activities involving

advocacy, policy support and capacitystrengthening, and adopting NationalExecution as a predominant mode of deliver-ing assistance.

n The changing environment for developmentcooperationIncreasing the need forcountry-based joint assistance strategies asemphasized by the Paris Declaration has alsoled to new aid modalities, such as directbudget support and sector wide approaches.This environment reinforces the shift toupstream activities.

1.3 WHAT ENTAILS RESULTS-BASEDMANAGEMENT IN UNDP?

Different organizations define results-basedmanagement in different ways, yet there is astrong common denominator among definitions.All reflect the underlying idea of learning fromempirical evidence based on past experience andusing that information to manage. TheOrganisation for Economic Co-operation andDevelopment-Development Assistance Committee(OECDDAC) Managing for DevelopmentResults Source Book10 puts it well:

Results-based management asks managers toregularly think through the extent to which theirimplementation activities and outputs have areasonable probability of attaining the outcomesdesired, and to make continuous adjustments asneeded to ensure that outcomes are achieved.

For results-based management to be successful,organizations need to develop and nurture aculture of results where enquiry, evidence andlearning are considered essential to goodmanagement. The use of results information inmanaging is usually seen as the main aim ofintroducing results-based management. In results-based management, managers are expected to:

9 UNDP, The Multi Year Funding Framework Report by the Administrator, UNDP Executive Board, DP/1999/CRP 4,January Session 1999.

10 OECD and World Bank, Emerging Good Practice in Managing for Development Results, First Issue, Source Book, 2006,p 9, available online at http://www.oecd.org/dataoecd/35/10/36853468.pdf.

C H A P T E R 1 . I N T R O D U C T I O N6

n Understand why the programme and projectsare believed to contribute to the outcomessoughtthe theory of change.

n Set meaningful performance expectations/targets for key results (outputs and outcomes).

n Measure and analyze results and assess thecontribution being made by the programmeto the observed outcomes/impact.

n Deliberately learn from this evidence andanalysis to adjust delivery and, periodically,modify or confirm programme design.

n Report on the performance achieved againstexpectationsoutcomes accomplished andthe contribution being made by theprogramme, i.e. what difference it is making.

When results-based management was introducedin UNDP, it was seen as involving all the abovefeatures, and the importance of a culture ofresults was well recognized. This is evident in aseries of notes produced in 2000.11 A longertreatment of these issues is given in Annex 6.

An important aspect of UNDP is the nature ofworking in a multilateral and decentralized

setting, where work is planned and managed atthe country level in response to diverse countryneeds yet done so within a global corporateenvironment that provides technical andmanagement support. Arrangements need to bemade so that planned results clearly respond tocountry needs yet are focused within a corporateframework that enables UNDP to add valuewithin its areas of competence. The decision onthose areas of competence is an issue for theExecutive Board, not an element of the results-based management system.

The challenges in implementing results-basedmanagement in an organization are many (seeBox 1).12 Perhaps key is, as the UNDP Overviewand Principles document noted,13 the importanceof emphasizing management and learning overreporting and systems, in order to foster a cultureof performance. Developing results frameworks,measuring results and reporting results in anorganization clearly will involve systems. If aculture of performance can be developed, thenthe main purpose of results-based managementwill not be lost. But without strong efforts todevelop and support such a culture, the systemsbecome the dominant feature. Senior managers

The World Bank 2006 Annual Report on Operations Evaluation reviewed progress with managing forresults.14 The report found:

n The World Bank has instituted policies and procedures to manage better for results.

n These have not yet translated into improved practices at the operational level.

n World Bank managers and operational staff struggle to link goals to operations.

n Performance indicators are often inadequate.

n Many staff are unclear about how to use performance information in their day-to-day work.

n World Bank culture acts as a disincentive to managing for results.

These findings resonate with the challenges faced by UNDP, described in this report.

Box 1. Results-based management is a challenge faced by other development partners

11 UNDP, RBM in UNDP: Overview and Principles, 2000; UNDP, RBM in UNDP: Technical Notes, 2000; UNDP,RBM: Concepts and Methodology, 2000.

12 See for example Mayne J, Challenges and Lessons in Implementing Results-Based Management, Evaluation, 2007,Volume 13, Issue 1, 89-107.

13 RBM in UNDP: Overview and General Principles, p 6, available online at http://www.undp.org/eo/documents/methodology/rbm/RBM-Overview-GP.doc.

14 World Bank, Annual Report on Operations Evaluation, IEG, 2006. Quote from website, www.worldbank.org/IEG.

C H A P T E R 1 . I N T R O D U C T I O N 7

have a special role to play in fostering this climateof results through clear leadership anddemonstrating that results and results manage-ment do matter. Reviews of experience withresults-based management show that:

Leadership support for results-based managementreforms is important. Without strong advocacyfrom senior managers, results-based manage-ment systems are unlikely to be institutionalizedbroadly or effectively within an agency. Leaderscan send strong messages of support for results-based management to their staff by giving speeches,sending out agency-wide notices, participating inresults-based management-oriented workshops,providing adequate budgetary support, etc. 15

Not everyone considers results-based managementa good system. Critics of using results-basedmanagement in public management point to anumber of aspects of public-sector life thatmitigate against a rational approach to managing.A case is often made that trying to manage bynumbers in a political context is, at best, unreal-istic and can be dysfunctional. In trying to setclear and concrete objectives and targets, politicalscientists argue that results-based management canconflict with the need to keep objectives suitablyfuzzy in order to gain widespread support.

This is true to some extent, but in the end,UNDP has to make choices about fundingspecific programmes consistent with nationalpriorities. Clarity in objectives can only help theirdesign and delivery.

Other critics of results-based management arguethat many of the developmental results sought byUNDP and other public-sector organizationscannot be measured. As a result, results-basedmanagement forces measurement and reportingof other less important results, especially outputs.

But many, if not most, results sought can bemeasured, especially if the evaluation teamconsiders measurement in the public sector to bea means of reducing the uncertainty about whatis happening, rather than definitively provingsomething. Flexibility in measurement approacheswould allow a wide variety of means to be used toincrease understanding about the performance ofa programme from different perspectives.

Focusing on any set of performance indicatorscan distort behaviour as people work to reachtargets. Arguably, this is a characteristic of thedrive for resource mobilization and delivery inUNDP. While this is a real problem, there aremany ways to counter this tendency, such asfocusing on outcomes not outputs, reviewingmeasures regularly, using a balancing set ofindicators, and developing indicators in aninclusive manner.

There are legitimate concerns over results-basedmanagement, and organizations should be awareof the possible downsides of implementingresults-based management. More details andspecific references are given in Annex 6. It isimportant to take these concerns into account indeveloping and managing the results-basedmanagement regime.

The Overview and Principle document foresawthis potential problem when it noted that RBM [results-based management] is a learningprocess evolving over a considerable period of time and incorporating flexibility to makechanges as experiences are gained.16 Results-based management is at the same time conceptuallyquite simpleseeking and using results informationto assist management and accountabilityyet asignificant challenge to implement in organiza-tions since it does require culture change andpersistence. It is a journey, not a destination,requiring ongoing attention and commitment.

15 Binnendijk A, Results-Based Management in the Development Cooperation Agencies: A Review of Experience,Background Report, DAC OECD Working Party on Aid Evaluation, Paris, France, 2001, p 134. Available online athttp://www.oecd.org/dataoecd/17/1/1886527.pdf.

16 RBM in UNDP: Overview and General Principles, p 5, available online at http://www.undp.org/eo/documents/methodology/rbm/RBM-Overview-GP.doc.

C H A P T E R 1 . I N T R O D U C T I O N8

1.4 STRUCTURE OF THE REPORT

This report is structured in five chapters.Following this introduction, Chapter 2 provides anoverview of results-based management and examineshow it was introduced to UNDP and what tools

and systems were developed. Chapter 3 presentsthe main findings from the country visits andinterviews at UNDP Headquarters. Chapter 4interprets the findings and tries to explain theprogress that has been made to date. Chapter 5includes conclusions and recommendations.

n The central feature of a results-based management system is managers using information to guidemanagement decisions.

n UNDP documents show that the importance of change to a culture of results was well recognizedwhen results-based management was introduced.

Key points

C H A P T E R 2 . R E S U L T S - B A S E D M A N A G E M E N T I N U N D P 9

2.1 THE EVOLUTION OF RESULTS-BASED MANAGEMENT IN UNDP

The history of results-based management inUNDP can be traced to the mid 1990s. Manypeople in the organization see the process as alogical evolution of earlier initiatives, such asprogramme planning. For the purposes of thisevaluation, the Administrators Annual Report of1997,17 which calls for the establishment of anoverall planning and results management systemin UNDP is taken as a defining point. Box 2highlights some characteristics of the approachto results-based management taken by special-ized funds and programmes.

The Administrators Annual Report clearly statesthe intention to develop plans that would bealigned with the budget for the 1998-1999biennium, and the organization would be able to

manage results against planned goals and targetsand with the appropriate resources assigned toachieve those results.18 However, there is nosingle document in which the organizationdescribed how its overall approaches and systemswould be changed to institutionalize a results-based management culture, or how the organiza-tion would measure the degree to which theintended changes in management practice hadbeen achieved. The various tools and systems thatcomprise results-based management emergedover several years and are set out in detail inTables 1 and 2 in Annex 5.19 The main buildingblocks are summarized in Table 1.

UNDP has invested heavily in the developmentof new results-based management relevantsystems between 1998 and 2007 and the period is characterized by a high degree of change.

Chapter 2

RESULTS-BASED MANAGEMENT IN UNDP

Year Key Event

1998-1999 Strategic results frameworks piloted then adopted across all country programmes

2000 First Multi-year Funding Framework (MYFF); First Results Oriented Annual Report (ROAR);Balanced scorecard introduced across all country offices

2002 Revision of Results and Competency Assessment (RCA); Handbook on M&E for Results;Reorganization of Practice Areas to match the MYFF

2004 ATLAS (Enterprise Resource Planning Tool) introduced

Table 1. Key events in the development of results-based management

17 Annual Report of the Administrator, DP/1997/16/Add.7, 1997.18 Ibid, p 36.19 In the survey conducted for this evaluation, staff identified the following systems to be part of results-based management:

MYFF, CPAP results framework, annual work plan targets, ROAR, CO balanced scorecard, outcome evaluations,project evaluations, RCA performance targets and assessment, executive snapshot, Global Staff Survey results, partnersurvey results, and ATLAS.

C H A P T E R 2 . R E S U L T S - B A S E D M A N A G E M E N T I N U N D P1 0

Within the UNDP group (UNCDF, UNV and UNIFEM), organizations have responded to results-based management indiffering ways. The text below reflects interviews held with the funds and a short review of documents onmanaging for results within their organizations.

UNCDFA specialized fund within UNDP. For the first time, UNCDF is being incorporated in the same StrategicPlan as UNDP (for the period 2008-2011). In the recent past, UNCDF has developed a strategic view of the Fundscomparative advantage in the context of the One UN vision. This has confirmed the Funds focus on two areas ofwork, namely Decentralization and Local Development and Microfinance/Inclusive Finance. There is a conscioussense of wanting to be leading reform, not catching up.

The Fund has a policy of trying to harmonize with UNDP and agree upon joint indicators and targets.20 Internaldocuments and interviews suggest that the Fund is looking at value-for-money questions, such as What are thecosts of achieving their targets?The Fund appears to be using results information: comparing countries, looking forbest practices, tracking new issues that arise, and figuring out where to investall examples of a results culture.A proposal to move towards supporting national programmes and sector-wide approaches as an aid modality wascirculated in February 200721 and has influenced the formulation of outcome indicators for the Strategic Plan. Thepaper describes the two focal areas of the programmes in a results framework that has five core results and 15outcomes groups. The outcomes are realistic, specific and measurable. In support of these objectives, the Fund hasset out planned intervention maps that show how financial and technical support from the Fund can lead toProgramme Purpose (Outcomes).

UNIFEMThe United Nations Fund for Women.22 In 1998, UNIFEM adopted several standardized tools to putresults-orientation into practice including the mandatory requirement of developing logical frameworks for allprogrammes, and the revision of the guidelines for periodic reporting to focus on results. Terminology is notidentical to UNDP but draws on their definitions and others such as the OECD-DAC and specialist organizationsincluding Save the Children. UNIFEM shares some characteristics with UNDP. UNIFEMs work in empowering womenand promoting womens human rights means that processes or how things are done can be as important as thefinal results of a project or programme. In addition, much of the soft assistance that UNIFEM provides in terms ofadvocacy, policy advice/dialogue, and facilitation/brokerage of information, etc. is geared towards creating orconsolidating processes that can facilitate womens empowerment.

An independent review in 2002 identified degrees of ambiguity and/or fragmentation among UNIFEM practicesand policies in terms of what, when and how progress should be tracked. This led to simplification of the existingresults framework, but the main change came with the 2004-2007 MYFF. The strategic results framework has justfour goals and four outcomes (the outcomes cut across all four goals) that are realistic, specific and measurable.These are accompanied by a set of indicators that are to a large extent linked back to the MDGs and identifyinternational sources such as UNAIDS and UNFPA. An evaluation of the programme was underway during fieldworkfor this evaluation.

UNVThe United Nations focal point for promoting and harnessing volunteerism for effective development.UNV has possibly been more closely allied to the UNDP approach than the other two organizations. The secondMYFF influenced how UNV reports results, as the annual reports to the Board structure discussion around the fivegoals found in the MYFF.23 UNV has been involved in both reporting against the MYFF and developing the newstrategic plan, where their main concern was to ensure that civic engagement was included as one of the operationalprinciples for development effectiveness. One challenge has been that UNV cannot report against significantaspects of its work (i.e. promotion of volunteerism) because the MYFF does not cater for that as an objective.

In 2005 and 2006, UNV developed its own business model and supporting results framework, which have beenaligned with the MYFF and were approved by the June 2006 UNDP/UNFPA Executive Board. They were developed ina participatory manner with staff and partners. The UNV results framework is currently designed at the corporatelevel and needs further work to ensure consistent understanding of the results across staff within the organizationand internal coherence. The focus is now shifting to how to operationalize the framework at the business unit andproject levels, which will include definition of suitable indicators and strengthening analysis of how UNV contributesto achievement of results. Major technical challenges will include the fact that UNVs contribution is mostly at thecommunity level and therefore not ordinarily picked up by national monitoring systems and the fact that UNVvolunteers work within projects administered by others, making it more difficult to identify UNVs contribution.

Box 2. Managing for results in associated funds and programmes

20 Proposed Partnership Framework for UNDP and UNCDF, DP/2007/11; Progress Report on the UNDP-UNCDFStrategic Partnership, DP/2007/34.

21 UNCDF, Moving to Sector-wide Approach for Decentralization and Local Development and Inclusive FinanceA Strategic Shift for UNDP and UNCDF in a Direct Budget Support Environment, 2007.

22 The work of UNIFEM is mandated through two international agreements: the Beijing Platform for Action resultingfrom the Fourth World Conference on Women in 1995, and the Convention on the Elimination of All Forms of Discrimination

C H A P T E R 2 . R E S U L T S - B A S E D M A N A G E M E N T I N U N D P 1 1

Annex 5 Table 1 shows that at least one keyinitiative relevant to results-based managementhas been launched across the organization everyyear. Each of these initiatives has demanded thatcountry offices do at least one of the following:

n Participate in the design of new systems

n Change their business processes to reflectnew systems introduced

n Change the information that is reported tocorporate level on their performance

2.2 INTENDED EFFECTS ON THE ORGANIZATION

In the absence of a single statement of strategythat results-based management can be evaluatedagainst, the approach taken here is to identify thefollowing various themes of reform as a structurefor analysis: setting strategic goals, aligning resultsand resources with those goals, monitoring forresults, adjustment and learning, and evaluationand accountability.

2.2.1 SETTING STRATEGIC GOALS

The intended purpose for setting strategic goalshas been to allow greater focusing of theprogrammes at the country level. This has beenconsistent during the past 10 years, as reflected inthe quotes below from 1997 and 2007:

To be effective, UNDP cannot attempt to doeverything, even within its SHD [sustainablehuman development] framework. Given thediversity of national situations in programmecountries, achieving focus within the frameworkmust be accomplished primarily at the countrylevel. While respecting the need for country-level

flexibility, broad parameters and corporatestrategic objectives must be established globally tomaximise the capabilities, impact and substantiveaccountability of the organisation as a whole.24

The Strategic Plan, 2008-2011, seeks to takeresults-based management a step further byproviding an instrument that: (a) clearly articu-lates UNDP priorities, objectives, targets andperformance indicators; (b) creates a solid basisfor internal resource allocation; and (c) sets astronger platform for comprehensive resultsmanagement. ... This simplified framework willincrease UNDPs focus, clarify its areas ofcomparative advantage, and facilitate themeasurement and reporting on results. For eachfocus area, UNDP will spell out its key resultsareas and outcomes, with a view to furtherstrengthening alignment.25

These quotes clearly recognize that the focus ofprogrammes emanate from country-level needs,yet the instrument to implement this focus wasthe global strategic framework. This creates atension between corporate and country offices,and an effective results-based managementsystem has to balance these conflicting pressures.Annex 6 addresses this issue in more detail undera note on working in a decentralized structure.

The MYFFs, and presumably the new StrategicPlan, are also reported to have a prominentsecondary role in communicating with externalstakeholders, as a concise explanation of theprogramme. Intended uses include addressing:

n Questions from the differing ExecutiveBoard constituencies over which goalsUNDP should focus upon. For example, the

Against Women (CEDAW), known as the women's bill of rights. The spirit of these agreements has been affirmed bythe Millennium Declaration and the eight MDGs for 2015, which combat poverty, hunger, disease, illiteracy and genderinequality, and build partnerships for development. In addition, Security Council resolution 1325 on women, peace andsecurity is a crucial reference for UNIFEM's work in support of women in conflict and post-conflict situations.

23 UNIFEM, How Are We Doing? Tracking UNIFEM Progress in Achieving Results for Management and Learning,A Briefing Note, 2002; UNIFEM, Multi-year Funding Framework 2004-07, DP/2004/5.

24 Annual Report of the Administrator, DP/1997/16/Add.7, 1997, p 30.25 From Frequently Asked Questions about the Strategic Plan, internal guidance on UNDP intranet site concerned with

development of the new Strategic Plan.

C H A P T E R 2 . R E S U L T S - B A S E D M A N A G E M E N T I N U N D P1 2

fact that the MYFFs and Strategic Plan, tosome degree, reflect where the demand forUNDP services is, allows the organization towork in areas that some Executive Boardmembers consider sensitive. The clearestexample quoted has been the use of theMYFFs in supporting UNDPs increasedfocus in the governance area.

n Concerns within the wider UN family overwhether UNDP was moving into areas thatwould be better served by other organiza-tions within the United Nations.

n Donors questions about the value addedof UNDP and, accordingly, a need todemonstrate results to donors. This wasparticularly important given the relativedecline in UNDP funding during the 1990sand thus the need to demonstrate to donorsthe value of increasing their commitments tothe organization.

2.2.2 ALIGNING RESULTS

Within the broad goals set within the MYFFsand the Strategic Plan, UNDP has embarked ontwo (until recently) parallel strands of work relatedto definition of results by the organization.

The first has been the move at the countryprogramme level from managing project inputsto managing a portfolio of projects and otherUNDP support26 to deliver at the outcome level.This move started in 1998, with the piloting ofstrategic results frameworks, which werestructured around delivery of outcomes, in alimited number of country programmes. Thistool was then rolled out across all countryprogrammes in early 1999. All those interviewedduring this evaluation who were involved indevelopment of strategic results frameworks statedthat the intended purpose of the tool was to fostera strategic management approach at the countryprogramme level, in which programme managers

would clearly be aware of how they expectedUNDP projects and other support to contributeto delivery of the agreed outcome. This intentionis also clearly set out in the quote below:

A further general lesson that emerged was theimportance of stressing management overmeasurement. The fundamental goal of results-based management is to improve developmenteffectiveness, which requires helping managers tobetter manage. In comparing RBM [results-based management] systems, the distinction issometimes made between managing by resultsand managing for results. The former is princi-pally oriented towards accountability andexternal reporting; the latter focuses on a cycle ofplanning, periodic performance and organiza-tional learning. In implementing RBM,UNDP made a deliberate decision to emphasisemanagement and learning. RBM mustexplicitly aim at changing the way the organi-zation is managed, fostering a strategic orienta-tion and culture of performance. Improvedexternal reporting was approached as veryimportant, but a secondary benefit.27

Support for a strategic approach at the county levelhas also come from reforms to the UNDAF process,which have started to work through UNDP countryprogrammes in recent years. The introduction ofthe UNDP CPD and CPAP, using a formatcommon to the Executive Committee (ExCom)agencies,28 has helped to clarify the alignment ofUNDPs country programme with nationalpolicies and harmonization with partner UNorganizations. In particular, joint programmingfor the UNDAF provides a stimulus for UNDPto ensure that its programmes reflect strategicareas from the MYFF and do not conflict withthose of other UN organizations.

The second strand of work has been theintroduction and use of the balanced scorecard in2000. This tool was originally introduced for

26 Advocacy, policy dialogue and institutional strengthening, and field presence.27 UNDP, Development Effectiveness Report, 2000, p 23-24.28 The ExCom Agencies are UNDP, UNFPA, UNICEF and WFP.

C H A P T E R 2 . R E S U L T S - B A S E D M A N A G E M E N T I N U N D P 1 3

monitoring implementation and the results of theinternal management reforms proposed in theAdministrators Business Plans, 2000-2003,which were intended to drive cultural changewithin UNDP, as summarized in Table 2.29

These changes complement results-basedmanagement and support an environment thatfocuses on results.

2.2.3 ALIGNING FUNDING

The concept of results-based management assumesthat resources will follow resultsin other words,as results are aligned with goals, resources wouldbe managed to achieve those results. In practice,the extent to which alignment with goals andresults from the results-based managementsystem was intended to influence the allocationof financial resources was severely limited. This

partly reflects the view of significant constituencieswithin the Executive Board that funding shouldprimarily reflect needs rather than results. Majorsources of funds during the period under evalua-tion are shown in Table 3.

The sources and uses of funds in UNDP at thecountry office and corporate levels is complex, withlimitations on management flexibility. The maindistinction is first between core and non-core.The targeting and allocation of core resources ismanaged as a resource supply to programmes andoperations. Shortfalls in funding to achievedevelopment results have to be made up by extra-budgetary income and non-core sources.30

Taking core funding first, there are three core pro-grammatic budgets:TRACs 1.1.1, 1.1.2 and 1.1.3. 31

Today Tomorrow

Project driven Policy driven

Process orientation Results orientation

Low-level specialized expertise Clear competency profile

Low knowledge-based capacity Innovative and information technology networked capacity

Risk aversion Risk taking

Introverted, sceptical of partnerships Outward looking, partnerships oriented

Cumbersome decision making Flexible and real-time decision making

Bureaucratic culture Merit-rewarding and initiative-driven culture

Weak management accountability Responsive leadership management

Table 2. Prospective cultural changes in UNDP

29 UNDP, The Way Forward. The Administrators Business Plans, 2000-2003, 1999, para 28.30 Assessment of Programming Arrangements, 2004-2007, paper presented to the Executive Board DP/2007/8, available

online at: http://www.undp.org/execbrd/word/dp07-8.doc.31 More commonly spoken of as TRACs 1, 2 and 3, for simplicity.

C H A P T E R 2 . R E S U L T S - B A S E D M A N A G E M E N T I N U N D P1 4

n The TRAC 1.1.1 budget represents theminimum level of resources targeted to beavailable for an individual programmecountry during a given financial period. It iscalculated in accordance with the board-approved distribution methodology, usingper capita gross national income and popula-tion as the primary criteria.

n TRAC 1.1.2 resources are in the firstinstance earmarked by region. These aresubsequently allocated by the r