Global Partnership ProgramGlobal Partnership ProgramEvaluation ChallengesEvaluation Challenges

Office of the Inspector GeneralEvaluation Division (ZIE)

1

Canada’s Global Partnership Program (GPP)Brief Program Overview

Program Rationale and MandateAt the 2002 Kananaskis Summit, G-8 Leaders committed toraise up to $20 B for a 10-year Global Partnership aimed to:• Address serious and immediate threats to the international security;• Prevent the proliferation of the stockpiles of chemical, nuclear and

biological weapons still available in Russia.

Canada demonstrated leadership by:• Committing CDN$1 billion over 10 years (2003-2013);• Establishing the DFAIT-led Global Partnership Program and funding

projects in five priority areas:• Chemical weapons destruction (CWD);• Dismantlement of nuclear submarines (NPS);• Nuclear and radiological security (NRS);• Redirection of former weapons scientists (RFWS); and• Biological non-proliferation (BNP).

2

GPP Evaluations

• Following Treasury Board Policies, all Gs&Cs programs are subject toregular evaluations;

• Seven (7) evaluations have been conducted since 2005 to assess therelevance, efficiency, cost-effectiveness and success of GPP and itssub-programs:– GPP Formative Evaluation (2006)- GPP Summative Evaluation (2007)- Summative evaluations of: CWD, NPSD, NRS, RFWS, BNP sub-

programs (2007)

• Overall findings and conclusions of the evaluations:– GPP is an outstanding program that has overcome many challenges.– Achievements to date are significant in all priority areas;– Results are consistent with intended outcomes.– Canada has made significant progress in project implementation, faster

than most of its partners, without sacrificing project quality;

3

Evaluation Focus and Approach

• Guided by the following pre-defined criteria:– Relevance (rationale for the program, alignment with

Canada’s international priorities and the needs of thebeneficiary, e.g. Russia)

– Results/Success (achievement of objectives andprogram impact)

– Cost effectiveness (program design, governancemanagement and delivery)

– Sustainability of results (lasting effects and benefits)

4

1. Evaluation Challenge:Evaluating Program Relevance in a Changing Context

• Evaluating Threat Reduction1. Geographic expansion and proliferation of the threat:

- New terrorist states are emerging in Central and South Asia, Africa;- Terrorist tactics are changing and proliferating.

What are the implications for the evaluation of a 10-yearinternational program that constantly adapts to change?

2. Changing perception of threat and increased focus on:- biological safety and security;- cross-boarder transportation.

How do we evaluate new program initiatives if these have notbeen specified in the initial Program documents (e.g. TBsubmission, RMAF)?

5

2. Evaluation Challenge:Evaluating Program Results and Success

• Attributing results to Canada’s contribution in a multinationalinitiative:– Donor countries are contributing different amounts– Donors want political/international visibility– Canada is the second largest donor after the US.

How do we evaluate/attribute results and success in thecase of multiple donors?

• Evaluating terrorist threat reduction• Terrorist tactics are changing• Weapons used for terrorist attacks are proliferating

How do we attribute success or failure to the program?

6

2. Evaluation Challenge (cont’d):Evaluating Program Results and Success

Reporting on performance targets in achanging political environment:• Achieving project/program results on time and within budget

when these depend on:• The full cooperation of Russia;• The timely transfer of contributions of other countries How do we measure program success and impact if the

scope of the problem is not always known?How do we assess the impact of changing Russian or

international priorities on Program success?• Ensuring sustainability of results:

Shall we be able to go back in 10 years and check thesustainability of our results?

7

3. Evaluation Challenge:Evaluating Program Cost-Effectiveness

Measuring the Effectiveness of Resource Allocation:

• GPP initial allocation and commitments based on 2002 priorities;• Ongoing re-prioritization and re-allocation of resources following

annual Program Reviews. How do we evaluate cost-effectiveness ?

• Making contributions to a country with a growing economic wealth:

How do we assess Russia’s need of international support ata time when its economic might and wealth are on the rise?

8

3. Evaluation Challenge (cont’d):Evaluating Program Cost-Effectiveness

Effectiveness of governance and deliverymechanisms:– GPP has employed a number of delivery channels and

mechanisms:• Canada-Russia bilateral Treaty;• Third-party delivery (piggybacking on partners projects); and• Contributions to International organizations (IAEA, EBRD, etc)

Should we seek a balance between efficiency and politicalvisibility?

How do we assess the effectiveness of Canada’scontribution to an international organization?

9

Further Evaluation Challenges:Impeding Factors to Success

• Factors affecting program implementation in Russia:– Slowness of the Russian bureaucracy;– Russia’s reluctance to provide access to necessary, but often

protected information;– Project sites situated in remote and difficult to access regions.

How do we develop feasible performance measures andtargets in a complex and changing environment?

• Taking into consideration the challenges for Russia:• Multiple donors conducting their own evaluations;• Understanding the difference between recipient audit and evaluation

Do we have the right to insist on receiving full access toprotected information of a sovereign country for thepurposes of program accountability?

10

From Challenges to Solutions

• Best practices in overcoming the challenges:– Do your homework, e.g.:

• Get familiar with the Program from its inception;• Conduct extensive research before starting an evaluation;• Consult with other donors / review other countries’ evaluations

– What else helps:• Knowing the country-specific context and political sensitivities;• Being aware of historical (legacy) issues;• Getting familiar with cultural and traditional aspects;• Exercising utmost flexibility, diplomacy and respect;• Sharing experience vs. conducting inquiries;• Treating Russia as a partner and not only as a beneficiary;• Speaking the language

11

Lessons Learned and Conclusions

Major Findings and Recommendations

GPP has generated impressive results in the firstfive years, leading to overall reduction of WMDproliferation threat;

Through GPP, Canada has demonstratedleadership among G8 countries in shaping andimplementing a major international initiative;

A longer-term vision for GPP is needed to ensuresustainability of results in Russia and the FSU;

Decisions on the future extension of GPP are tobe soon made: however, geographical expansion ofthe Program will depend on the political will andcurrent needs of the international community.

Despite the above mentioned challenges, GPP evaluations were conductedobjectively and professionally, providing reliable findings, analysis andconclusions regarding GPP relevance, cost-effectiveness, and results.

12

Challenges in Evaluating InternationalPrograms and Initiatives

Thank you / MerciThank you / Merci

Questions?

SPEAKING NOTES

CES CONFERENCE

EVALUATING CONFLICT PREVENTION AND PEACE BUILDING (CPPB) INITIATIVES

INTRODUCTION

- The events of September 2001 reminded the international community that civil wars and state failure in far flung regions of the world can have a direct impact on our own national security.

- It is understood that effectively responding to the challenges of states

in crisis or emerging from crisis calls for an integrated approach involving diplomacy, peace operations (military and civilian), institutional capacity building (governance structures), and development assistance.

- It will be obvious that delivering on these various components of

crisis response goes beyond the competencies (legislative and human) of any one department or ministry, but can only be achieved with the coordinated activities of many departments in a “whole-of-government” effort.

- To this end, the Government of Canada in 2005 created the

Stabilization and Reconstruction Task Force (START) under the authority of the Minister of Foreign Affairs and International Trade (DFAIT) with the mandate to ensure whole-of-government policy and programming coherence with respect to conflict prevention, peace operations, and post-conflict stabilization.

- The Government of Canada also created the Global Peace and

Security Fund (GPSF) with a budget of $ 500 Million over five years to support initiatives in the aforementioned domains.

EVALUATING CPPB ACTIVITIES

- In order to properly evaluate CPPB activities it is necessary to understand what, at a conceptual level, constitutes good programming practice. The Development Assistance Committee of the Organization for Economic Cooperation and Development (OECD-DAC) has been at the forefront of this endeavour, producing in 2007 its list of “Ten Principles for Good International Engagement in Fragile States.”

- All of the Ten Principles are important, but 4 are of particular

importance. These are: 1) Take context as the starting point [based on sound conflict analysis]; 2) Act fast and be flexible [change course in response to changes in the dynamics of the conflict]; 3) Stay long enough to give success a chance [conflict resolution takes a long time]; and, 4) Adopt a coordinated approach [WoG and inter-donor coordination].

- While all the OECD-DAC best practice principles can find a place

within the standard evaluation criteria of Relevance, Efficiency and Effectiveness, and Success/Results, how CPPB activities are assessed in reference to these criteria differ slightly from more conventional development/humanitarian programs.

a) Relevance

- Whereas as development programs are designed and objectives

formulated based on needs assessments, CPPB activities are designed and objectives formulated on conflict analysis i.e., identification of the “key driving factors” or “structural causes” of the conflict. This includes an examination of the nature of the grievance between the parties and their respective interests and capacities. Assessing the validity of this analysis is the first test of relevance.

- The second test of relevance relates to the “theory of change” that

guides the kind of intervention to be pursued. For example, a ‘theory of change” may espouse the view, given one’s understanding of the conflict, that strengthening civil society rather than directly engaging the leadership is the most effective way to effect desired change. Testing the validity of this theory, through examination of its actual

impact on the behaviour of parties and their respective capacities, provides a measure of the relevance of the theory.

b) Efficiency and Effectiveness

- There is a general consensus among evaluation practitioners of CPPB

activities that the criteria of efficiency and effectiveness are of subordinate importance to the evaluation issue of relevance. A project may be efficiently executed and effective within its defined sphere, but if the project does not positively impact on the dynamics of the conflict, then the project would have to be deemed ineffective.

- Notwithstanding that “value for money” may be difficult to assess

with CPPB activities, and hazardous to apply rigorously, efficiency and cost effectiveness cannot be ignored and, in cases where speed is held at a premium [getting humanitarian of peacekeeping supplies on the ground rapidly], highly relevant to assessing success and impact.

- There are several other evaluation criteria peculiar to CPPB activities

which directly impact on efficiency and effectiveness, and ultimately success. These are - and they relate to the OECD-DAC principles cited earlier – Flexibility and Responsiveness and Coordination and Coherence.

- In a dynamic conflict situation, a high premium is placed on the

ability of donors to rapidly shift course in response to changes in the conflict. This calls for a degree of flexibility that is, in many ways, at odds with standard performance and results reporting logic, which tend to focus on linear causal relationships between outputs, outcomes and impacts (with fixed indicators) and do not capture feed-back loops or the effects of external factors.

- The other evaluation criteria of note – Coordination and Coherence –

is of paramount importance in assessing efficiency and effectiveness and, ultimately, success.

- As remarked at the outset, diplomacy and dialogue, justice and

security sector reform, institutional capacity building, peace operations, and development are beyond the competencies of any one ministry or department. Ensuring that initiatives in these areas

implemented by different actors are aligned and properly sequenced is of critical importance to effectiveness.

- Police training, for example, is likely to be of little consequence if

there is not a trusted and efficient court and correctional system in place. Demobilizing ex-combatants will likely lead to increased criminality if not complemented by a program of integration. Coordinating such diverse activities by different actors presents a huge challenge, not simply for donors among their own participating departments, but among donors themselves.

- There must also be agreement among donors on policy, because

policy informs programming, lest donors find themselves working at cross purposes and potentially aggravating the conflict. Again, given the different capacities and priorities of donors, this presents an enormous challenge.

- Within the Canadian context, there are a variety of obstacles to

effective WoG coherence and coordination in response to crises: • The priorities of departments may differ significantly from

those of OGDs. • Planning and budgeting cycles between departments are not

necessarily in synch with OGDs, including DFAIT, thus reducing flexibility to mobilize resources in response to a new or emerging crisis.

• Accountability and reporting, necessary for effective WoG management, remains stove-piped.

c) Success and Impact

- Assessing the success of CPPB activities is extremely difficult for the

following reasons:

• Indicators of progress and impact with respect to conventional development programs remain constant throughout the life cycle of the program, whereas the indicators of progress and impact used for CPPB activities change over time in response to the dynamics of the conflict.

• Quite often there is an absence of baseline or benchmark data needed to assess progress and attribute results to the intervention.

• Peace-building is a long-term endeavour where the dividends

from one’s investment may not be apparent for some time beyond the standard reporting period required by central agencies – success is measured in CPPB activities in incremental steps extended over a long period of time.

• Many CPPB activities do not produce well defined outputs or

outcomes, but intangibles such as changes in attitudes and relationships which often difficult and expensive to assess.

• Owing to the constellation of factors outside the control space

of a given intervention, which may undermine effectiveness, one must be cautious before dismissing the relevance of the intervention simply because “success”, however defined, has not been achieved within the stated time frame.

• CPPB activities are conducted in highly unstable and

threatening environments (when in Sudan several NGO workers had been kidnapped, which resulted in a cessation of operations), thereby demanding a higher tolerance for risk, and even failure, in a way that is foreign to central agency requirements.

• CPPB activities aim to alter the dynamics of a conflict, which

invariably empowers some while marginalizing others. This alteration in the power balance between parties can, in the short term, contribute to an increase in tension and hostility, but which in the long run may lay the foundation for compromise and accommodation.

• Because of the wide variety of factors impacting on the

dynamics of a conflict, establishing a causal connection/attribution between results and the intervention can be extremely imprecise.

- Notwithstanding the challenges of evaluating CPPB activities against logical framework and results-based models, both have their place in evaluating CPPB activities. Their relevance and utility would be greatly enhanced if they were to incorporate several of the evaluation criteria alluded to in this course of this brief presentation, namely: a description of the theory of change informing the intervention and assumptions relating thereto; situating the intervention within the life cycle of the conflict in order to more precisely identify immediate outcomes; adopting flexibility and responsiveness as a performance indicators; and, expressly recognizing coordination and coherence as key enablers for effectiveness and ultimately success.

TRANSPARENCY AND ACCOUNTABILITY IN EVALUATING CPPB INITIATIVES

- Both programmers and evaluators of CPPB activities must be highly sensitive to the impact both have on the parties to a conflict. As mentioned earlier, third party interventions invariably alter the power balance between these parties.

- For evaluators of CPPB activities, adhering to the cardinal principle of

“do no harm” can present a challenge in terms of honouring the principles of transparency and accountability. Where, for example, an evaluation finding singles out a particular party to the conflict as a “spoiler”, in particular if that party is a government or an arm of government, acknowledging this in a public report, even if it has a direct bearing on program performance and results, can have serious diplomatic ramifications.

- In short, evaluators of CPPB activities must, in the drafting of their

reports, exercise a degree of caution and sensitivity to the implications of their findings and recommendations that is uncommon with other forms of program evaluations.

CONCLUSION

- In sum, there are many challenges to evaluating CPPB activities that

distinguish it from other evaluation exercises, but these challenges are not insurmountable. While certain allowances are called for in

evaluating the performance of CPPB initiatives, the issues of relevance, efficiency and cost effectiveness and success are as pertinent to CPPB initiatives as any other intervention.