Considerations by KfW on Sustainabilityin Final Evaluations of FC Projects: Basic Considerations

Discussion Papers

FC Projects and Sustainability

33

I M P R I N T

Published by: KfW, Secretariat of International Credit Affairs

Edited by:KfW, Evaluation Department of FC

Frankfurt am Main, October 2003

KfW GroupPalmengartenstrasse 5-9

60325 Frankfurt am Main, Germany

Postfach 11 11 41

60046 Frankfurt am Main, Germany

Phone +49 69 7431-0

Fax +49 69 7431-2944

www.kfw.de

Press Department

Phone +49 69 7431-4400

Fax +49 69 7431-3266

Information Centre

Phone +49 180 1 335577

Fax +49 69 7431-64355

e-mail: info@kfw.de

Berlin Branch

Charlottenstrasse 33/33a

10117 Berlin, Germany

Phone +49 30 20264-0

Fax +49 30 20264-5188

Bonn Branch

Ludwig-Erhard-Platz 1-3

53179 Bonn, Germany

Phone +49 228 831-0

Fax +49 228 831-2255

CONSIDERATIONS BY KFW ON SUSTAINABILITY IN FINAL EVALUATIONS OF FC PROJECTS: BASIC CONSIDERATIONS

0. Summary 1

1. Preliminary Remarks 3

2. Final Evaluations, their Purpose and their Approach 3

3. Criteria for evaluating Project Success in FC 6 3.1. The relevant evaluation criteria 6 3.2. On defining sustainability 7 3.3. Determinants of the sustainability of projects 11 3.4. Considering the aspect of sustainability in final evaluations by KfW 13 3.5. Projects with limited sustainability requirements 15

4. The Informative Value of Final Evaluations 17

Annex 1: KfW’s rating categories for project evaluation 20

Annex 2: Individual aspects of project classif ication 21

0. Summary

(1) KfW carries out ex-post evaluations for all Financial Cooperation (FC) projects

that it manages. These take the form of final project evaluations for which the central

evaluation unit FZ E (Financial Cooperation Evaluation) is responsible. The aim of a

final evaluation is the systematic comparison of the actual project impacts that can be

observed at the time of the final evaluation with the expected project impacts at the

time of the project appraisal, with the stated project purposes and overall objectives

serving as the basis.

(2) The assessment of developmental success during the final evaluation is

principally based on the technical requirements and standards valid at that time.

These result from the relevant sectoral and cross-sectoral promotional concepts of

the Federal Ministry for Economic Cooperation and Development (BMZ) and the

partner country, sector-specific operational criteria for evaluation, as well as general

development-policy criteria and professional standards. Thus, changes in the

evaluation standards for determining the project impacts compared with the situation

at the time of project appraisal need to be accounted for (qualified target-

performance comparison).

(3) The altogether six performance ratings for evaluating project impacts make it

possible for a project to be classified as successful or not successful. In principle,

only projects that – on the basis of current requirements and standards – would

obtain a positive performance rating in a (hypothetical) renewed appraisal at the time

of the final evaluation should be classified as successful.

(4) In evaluating a project’s developmental success and assigning it to one of the

performance categories the following points are examined:

* whether the project purposes set for the project were sufficiently achieved

(aspect of effectiveness);

* whether the overall objectives set for the project were sufficiently achieved

based on the use of the produced goods and services (aspects of relevance

and significance);

* whether the project output is generated, used and marketed with sufficient

consideration of microeconomic and macroeconomic requirements (aspect of

efficiency).

- 2 -

(5) Within the context of a final evaluation at KfW a project is considered sustainable

if, after termination of the external financial, organizational and/or technical support

for the project activities, the project-executing agency and/or the target groups are

willing and able to successfully continue the project on their own for an acceptable

period of economic life – which depends on the project type - and if the overall

objectives and the project purposes can be achieved in the future as well. The

“acceptable period of economic life” is usually based on the customary economic

lifetime of the investment, taking into account the overall conditions specific to the

project (e.g. climatic or social-policy conditions, but also taking maintenance

measures with an investment nature into account). Here, relevant standards are to

be defined for each region, sector and target group.

(6) According to the evaluation experience that has been gathered from projects to

date both endogenous and exogenous factors are relevant for the long-term impact

of the projects. The determinants endogenous to the project involve, on the one

hand, the underlying target system, the project conception and the inclusion of the

project in the given political, economic, legal and socio-cultural conditions (including

the performance of the project-executing agency). On the other hand, they involve

the acceptance of the project by the political decision-makers, the target groups and

the local executing agencies. The determinants exogenous to the project mainly

include the policies of the partner country, the global economic conditions (which are

mainly shaped and influenced by the industrialized countries), technical progress and

exogenous shocks (such as natural catastrophes).

(7) In KfW’s final evaluations sustainability is not treated as an independent criterion

of evaluation but is instead accounted for during the assessment of the key criteria of

effectiveness, relevance/significance and efficiency. Since these criteria are not met

in equal measure, separate classifications are made in accordance with the six-step

scale of performance evaluation for the three key criteria of (sustainable)

effectiveness, (sustainable) significance/relevance and (sustainable) efficiency. The

three partial classifications are then combined to a final classification; depending on

the project, it is entirely possible that the partial classifications are weighted

differently. In addition, varying sector-specific minimum requirements may be defined

for all three key criteria, the non-fulfilment of which unavoidably leads to an overall

evaluation of “not successful.“

- 3 -

1. Preliminary Remarks

(1) KfW carries out ex-post evaluations for all Financial Cooperation (FC) projects1

that it manages. These take the form of final project evaluations for which the

central evaluation unit FZ E (Financial Cooperation – Evaluation) is responsible.

According to KfW guidelines2 the objective of a final project evaluation is “the

systematic recording of the intended and unintended project impacts to evaluate the

developmental success of the financed project.“ Here, the question of sustainability

frequently plays a decisive role in the evaluation of project success. Yet, it is not

always sufficiently clear how the aspect of sustainability is taken into consideration

during a final evaluation. For this reason, the manner in which this is done is to be

documented.

(2) This paper concentrates on several basic considerations on the sustainability

of projects. It mostly refrains from covering sectoral and specific project

particularities. These particularities will be handled in special papers on the key

promotional fields and sectors of FC (such papers are presently being prepared).

Therefore, this paper will remain relatively abstract. However, it provides the

fundamentals for ensuing papers. Without knowledge of the basic paper, it will be

difficult to fully understand the sector-specific reasoning.

2. Final Evaluations, their Purpose and their Approach

(1) The evaluation of the developmental success of a financed project is guided by

the project impacts at the time of the final evaluation – those that have actually arisen

and those that realistically continue to be expected. The aim is to determine the

developmental impact of the project concerned. Accordingly, a final evaluation

should not be carried out unless it is reasonably sure that a conclusive analysis of the

project’s developmental impact is possible. In KfW’s practical experience with final

evaluations, a period of 3-5 years after the final follow-up (which is carried out

following the physical completion of the project) is usually assumed; depending on

the project type (e.g. for projects in the financial sector), shorter time periods are also

possible.

1 A project comprises a number of project measures required to produce certain goods or services that are important in developmental terms. It is irrelevant whether these project measures are financed through FC funds, by third parties or the project-executing agency itself.

- 4 -

(2) The key element of a final evaluation is the comparison of the actual project impacts (actual values) that were observed at the time of the final evaluation with the

expected project impacts (planned values) at the time of the project appraisal on

the basis of the stated project and overall objectives. The project objectives relate

to the operational level of the project. They describe the sustainable and efficient

production, market supply and provision of certain target groups with goods and

services intended when the project capacity is put to use. In this way, they reflect the

directly targeted main impacts of the project. In contrast, the overall objectives

relate to the utilization level of the project. They describe the indirectly targeted

developmental impacts generated by utilizing the produced goods and services.

(3) In principle a project must be measured by the target level specified in the

project appraisal report. It may not be weighed down ex-post with further demands or expectations that were not included in the target system set for the project (i.e.

the project and overall objectives). Only then can a project’s intended impact be

compared with its actual impact. Yet, the adequacy of the target system needs to

be reviewed to avoid an excessively positive rating in the final evaluation because

the target level was set very low, or to avoid an excessively high target level that

could lead to an unfavourable evaluation of the developmental success of a project.

(4) The assessment of developmental success during the final evaluation is

principally based on the technical requirements and standards valid at that time.

These result from the relevant sectoral and cross-sectoral promotional concepts of

the Federal Ministry for Economic Cooperation and Development (BMZ) and the

partner country, sector-specific operational criteria for evaluation, as well as general

development-policy criteria and professional standards (‘state of the art’).

Consequently, changes in the standards of evaluation applied to assess the project

impacts in comparison with the situation during the project appraisal have to be

accounted for (qualified target-performance comparison). This affects in part the

use of the indicators of project success, applied to measure achievement of the

objectives, and in part also the consideration of general development-policy goals

valid at the time of the final evaluation (for example the consideration of

environmental impacts, even if they were not mentioned in the project conception). In

case the application of the requirements and measures initially set for the project

2 See KfW Handbook, Chapter 3.6.1.

- 5 -

planning and appraisal would in individual cases lead to a success rating that is

considerably higher than would result by applying current requirements and

standards, this issue must be treated in the respective final evaluation report. After

all, the classification of the results must be oriented towards current requirements

and standards.

(5) A general methodical problem with ex-ante and ex-post evaluations of individual

projects is the clear attribution of an impact to certain measures. The number of

factors on which a project's impact depends is large, and these factors influence one

another. For this reason an evaluation of project success must take into account that

the project's impacts depend heavily on the system in which it is embedded. On the

other hand, nearly every project also affects its environment and thus influences the

overall conditions specific to the project. Isolated project analyses that do not take

the specific project environment into account are therefore not meaningful since they

ignore key indicators of project effects.

(6) A key goal of bilateral and multilateral development cooperation is to contribute to

the build-up and reinforcement of structures that foster development in the recipient

countries. Accordingly, during the final evaluation it should be verified whether and

how the project under review contributes to building up and strengthening such

structures or is involved in such processes. However, this is not often the case owing

to the size of individual projects compared with the total volume of bilateral and

multilateral development aid and the need for such support. Thus, even though

individual projects often have only a limited scope a systematic approach is to be

applied which accounts for the project's embedment in its environment and for the

resulting opportunities for mutual influence between the project and its environment.

For this reason it needs to be ensured that a project ready for final evaluation is not

overburdened with unacceptably high expectations of system-changing effects that it

is unable to satisfy due to its size. In principle, though, one minimum requirement

must apply: the project under review must be embedded in a structurally beneficial

environment (e.g. it could be part of a promising priority area strategy with clearly

defined goals).

This, in turn, places the focus on the question of how the relevant environment is

determined (this relates to both cross-connections within the pertinent sector and

also to cross-sectoral relations) and how it differs from other components. This

question can only be answered individually for each project.

- 6 -

(7) In principle the performance evaluation also covers the project rationale itself –

i.e. the effectiveness of the causal relations assumed in the project planning – from

the perspective of the ex-post evaluation. This examination of the project rationale is

required especially for substantial deviations from the planning and/or unforeseen

project impacts (positive or negative). In addition, the analysis of such deviations

from the project planning is the basis for evaluating the quality of the planning,

project steering and the performance of the main stakeholders - which is equally

important during the final evaluation - as well as for gathering vital experience for

future projects.

(8) The altogether six performance rating categories to evaluate developmental

sucess (see Annex 1 for details) make it possible for a project to be classified as

successful or less than successful. In principle, only projects that – on the basis of

current requirements and standards – would obtain a positive performance rating

category in a (hypothetical) renewed appraisal at the time of the final evaluation

should be classified as successful.

3. Criteria for evaluating Project Success in FC

3.1. The relevant evaluation criteria

The evaluation of a project’s developmental success and its classification into one of

the six performance rating categories center around the following fundamental

questions:

(1) Aspect of effectiveness: Are the objectives that the project is meant to fulfil

achieved sufficiently via specific production and supply levels (the aspect of project

effectiveness)?

(2) Aspect of relevance and significance: Are the overall objectives that the

project intends to meet achieved sufficiently through the use of the generated goods

and services by final consumers/the target group as well as through the impacts

generated in this connection?

a) Was the project conception adequate to solve the problems, i.e. does the

achievement of the project objectives at all contribute to the achievement of the

overall objectives set for the project and of other development-policy goals

generally accepted and deemed important (for example ecological, gender and

poverty reduction goals) (question of developmental relevance)?

- 7 -

b) Were the project’s effects with regard to the aspired overall objectives felt to a

sufficient degree (question of developmental significance)?

(3) Aspect of efficiency:

a) Were and are project and overall objectives being achieved through the use of

funds in a manner appropriate in microeconomic and macroeconomic terms

(question of production efficiency)?

b) Do the use/marketing of the project output take microeconomic and

macroeconomic requirements into sufficient account (question of allocation efficiency)?

(4) Where undesired side effects have occurred, e.g. environmental, social or

socio-cultural effects (e.g. gender effects), are these tolerable?

3.2. On defining sustainability

(1) The aspect of sustainability – which is vital for evaluating a project - is one of the

most many-faceted terms in the international development-policy debate.3 Within the

context of a KfW final evaluation a project is considered sustainable if, after

termination of the external financial, organizational and/or technical support for the

project activities, the project-executing agency and/or the target groups are willing

and able to successfully continue the project on their own for an acceptable period of economic life – the length of which depends on the project type - and if the project and the overall objectives can be achieved in the future as well.

(2) Thus, the request for sustainability has a three-dimensional character:

a) the dimension of time, i.e. the period of time (“acceptable period of economic

life“) for which the project’s continuation must be ensured in order for it to be

considered sustainable. The “acceptable period of economic life” is usually

based on the customary economic lifetime of the investment, taking into

account the overall conditions specific to the project (e.g. climatic or social-

policy conditions, but also taking maintenance measures with an investment

nature into consideration). Depending on the project type and sector, this leads

to a very broad range of “acceptable periods of economic life” in practice. Thus,

3 The term “sustainability“ was first used in forestry: only so many trees should be felled for wood and taken out of the forest as can grow in the respective area. “Sustainable“ means “lasting,“ “having long after-effect” and “permanent” for the purpose of “continued effect.”

- 8 -

the requirements of the economic life of efficient infrastructure projects

frequently have a different dimension of time than measures to promote health

care or the prevention of AIDS. Consequently, the “acceptable period of

economic life” must be defined more precisely for each sector and project in the

sector papers still to come.

b) the dimension of autonomy, i.e. to what degree recourse to external assistance

is acceptable and from whom such assistance may be accepted (from donors,

from the State, from population groups that are not part of the target group, etc.)

in order to ensure the ability of the project to function and to maintain its

intended impacts. Here, the main focus is on the organizational, financial and

technical performance of the project-executing agency on-site. For the financial

performance, whether and in what amount subsidies for the executing agencies

are acceptable without jeopardizing their autonomy must also be taken into

consideration. In case of market failure or market rejection – the traditional

manner of legitimization of governmental intervention in the economic process –

the project-executing agency is almost always dependent on state subsidies.

Yet, this is not a problem in and of itself unless the respective activity should, in

principle, not be carried out by the State. Rather, what is relevant in this context

is the reliability of the governmental subsidies as a prerequisite for

sustainability.

In numerous countries such governmental subsidies, even if they are tolerable

for regulatory-policy concerns (e.g. in case of positive externalities or

distributional-policy goals), cannot be financed by the country without external

assistance. This affects many of the least developed countries (LDC) in

particular. In such a situation, for the projects concerned there can be no

mention of sustainability in the sense of autonomy. Nevertheless, supporting

precisely these projects and countries can be extremely important from a

development-policy perspective. In such cases, however, the risks arising from

the country’s dependence on donor subsidies should be explicitly described and

assessed.

c) the dimension of the quantity and quality of the project impact, i.e. the

minimum degree to which the relevant effects must continue to be felt after a

certain period of time, also if the project is carried on independently, in order to

- 9 -

make a positive contribution to the achievement of the project and overall

objectives. Thus, project results may also include changes in behaviour.

Yet, the operational determination of generally accepted standard values for these

three dimensions is confronted with insurmountable obstacles owing to the

heterogeneity of the FC projects. Furthermore, it does not seem to make sense.

Instead, relevant standards are to be defined for each region, sector, and target

group. They must require a minimal relative improvement over the initial situation

(which would, however, have to be documented clearly).

(3) Thus, the term 'sustainability' within the context of a final project evaluation has a

different meaning than within the context of sustainable development. Text box 1

illustrates the differences.

In connection with a final project evaluation, sustainability refers to the sustainable

success of evaluated projects, meaning the continuation of project impacts. The

projects’ ecological compatibility – if it is not included in the explicitly stated target

system – can be assessed together with the undesired side effects and thus has

only an indirect effect on project sustainability.

Text box 1: The three levels of sustainability In the ongoing development-policy debate the term ‘sustainability’ is used on three different levels: a) The sustainability of projects refers to the continuation of the project impacts, although they may

vary considerably in their substance depending on the project. Therefore, it is better to use the term ‘sustainable success.’

b) The sustainability of the use of natural resources is given if the resources are not wasted but instead maintain their substance and usefulness (high degree of sustainability) or adequate set-off is offered (poor sustainability). The essence of this kind of sustainability differs greatly from the first one mentioned: A project can be sustainable without involving the sustainable use of resources and, vice versa, natural resources can be used sustainably without a project even being implemented at all.

c) Although sustainable development entails the sustainable use of resources, it goes far beyond that: Apart from the development of the natural resources and the environment overall, it also covers economic and social development and includes practically every conceivable level of development.4

(4) The sustainable success of projects basically breaks down into two

complementary elements that can be designated as internal and external project sustainability:

a) Internal project sustainability is derived from the sustainable achievement of the project objectives. In this context sustainability refers to the durability of the measure in comparison with the expected useful economic life of the

4 However, this runs the risk that the term ‘sustainability’ is often used arbitrarily and, in an extreme case, will

simply degenerate into a hollow word.

- 10 -

project and its probable resistance or adaptability to external shocks, changing or incorrectly estimated conditions and unforeseen internal deficiencies of the project that do not become evident until it begins

operating. Projects that – for whatever reason – fail early on during a period of

time deemed adequate or that lose a substantial amount of their substance

(example: because of a lack of road maintenance a road connection

constructed as an all-weather road can only be used during the dry season) are

not sustainable and are thus failures. Therefore, this form of sustainability refers

mainly to the effectiveness and efficiency principle.

b) External project sustainability refers to the sustainable contribution of the project objectives to the achievement of the overall objectives. The aim

here is to determine whether the causal relation between the achievement of

the project objectives and that of the overall objectives on which the project

rationale is based is of a long-term nature. Especially for projects that go

beyond narrowly defined individual measures, the types of spillover effects

and structure-building effects that a project has on its environment are vital.

For example, was the organizational efficiency of the partner strengthened on a

long-term basis (i.e. beyond the period of external support) or were certain

changes in the behaviour of the target groups with longer-lasting effect brought

about? In such cases FC requires of itself (and justly so) that it contributes

(significantly) to solving problems with as much broad-scale effect as possible,

and that these contributions be recognized as particularly significant for a

specific development strategy. FC aims to generate structural impacts,

strengthen developmental impulses and encourage socio-economic

modernization processes of which longer-term, noticeable improvements in

people's perspective on life are expected. As a result, external project

sustainability, which is being discussed in this section, chiefly relates to the

relevance and significance principle.

A project has sustainable development success in this sense of the term if it

• has direct significant impacts on the overall development-policy objectives defined during the project appraisal, or

• if it has a strong capability to produce sectoral, regional or national spill-over effects (aspect of broad-scale impact), or

• if it contributes to structure-building, i.e. if it supports important structural reforms on the institutional level or contributes to networking among institutions, the State and non-governmental stakeholders, the economic

- 11 -

sector and its environment as well as sub-national, national and supranational levels of politics, or

• if it has the ability to serve as a model, i.e. can be copied by partners or German Development Cooperation (DC) actors in other sectors, regions or countries.

Both internal and external project sustainability are necessary in order to attain

sustainable project impacts.

(5) The longer a project's basic economic life (for many investments in physical

assets, in particular in the field of material infrastructure, an economically relevant life

of 15-25 years is assumed), the more evident the sustainability risks become. This

not only involves the risk that internal project sustainability risks resulting from

insufficient project maintenance, for example, will take effect but it also affects the

ability to react to exogenous shocks. Insufficient internal project sustainability is probably more likely to be deemed a planning deficiency in FC than insufficient external project sustainability.

3.3. Determinants of the sustainability of projects

(1) The experience with evaluations gathered thus far indicates that factors both

endogenous and exogenous to a project are relevant for its sustainability.

(2) Regarding the determinants of sustainability endogenous to the project, the

following aspects are particularly significant:

a) The target system on which the project is based must be clear, consistent and

empirically verifiable. Therefore, there must also be target congruency between

KfW and its partners in the recipient country.

b) The project conception has to be compatible with the given political, economic and legal conditions in order for it to be applied at all (without

making a taboo of them). The expected possibilities for the project to influence

these conditions must be realistically feasible.

c) The project must become part of the local socio-cultural system for it to be

accepted by the political decision-makers, the target groups and the executing

agencies on-site (idea of ownership). For this the project must satisfy an urgent

need of the target groups and offer visible relief. The consensus that is so

vital for acceptance of a project is made possible primarily through the early and

extensive participation of partners and target groups in the definition and

preparation of projects. A further important factor is the dialogue with those who

- 12 -

seem to or who definitely do suffer a disadvantage initially as a result of the

project.

d) The institutional efficiency of the project-executing agency responsible for the

project must be satisfactory for the project conception; the capabilities of the target groups in terms of operation and maintenance of constructed plants or

installations should not be overly strained. Ideally, an executing agency should:

I) be financially viable; II) have adequate and sufficiently qualified, competent personnel; III) offer an adequate organizational and operational structure for the intended

operation; IV) have sufficient autonomy against tight political control; V) assign its management broad responsibility.

To determine the degree to which these qualities are available, whether

appropriate flanking measures to strengthen the executing agency were

successfully applied during project implementation, and whether identified

deficiencies of the executing agency, if any, imperil the project impacts, a

careful analysis of the executing agency during the final evaluation is essential.

In this connection reference is made to KfW's internal guidelines, which

describe the main aspects of an analysis of a project-executing agency.5

e) A project’s commercial viability must be guaranteed. However, it needs to be

taken into consideration whether a long-term - in extreme cases permanent -

need for subsidies was already noted when the project conception was being

drawn up. Such a need for subsidies is admissible under certain circumstances

(e.g. if the market has failed or in case of market rejection, up to but not

exceeding the subsidy amount). For this reason, when the request for

commercial viability is reviewed, the volume of the subsidies arranged and committed ex-ante out of the state budget, from abroad or from private

sponsors must be accounted for.

f) The project may not have any intolerable environmental or negative gender impacts.

(3) With respect to the determinants of sustainability exogeneous to the project, the following aspects are particularly relevant:

5 Base-Line Appraisal Tool (BLAT): Basic grid for the evaluation of the performance of project-executing

agency, Jnuary 22,2003

- 13 -

a) The policies of the partner country: The success of development projects is

highly dependent on whether the government of a country and the target group

support these projects even after the external financial contribution has come to

an end. Political instability such as unrest, revolutions and war or frequent

personnel fluctuations at the head of government can jeopardize a project's

long-term impacts just as much as misguided national economic policy.

b) External factors: Natural catastrophes in the form of droughts, floods and

earthquakes can put a project’s sustainability at risk just as much as global economic conditions – which are chiefly defined and influenced by the

industrialized countries – and also technological progress.

(4) These determinants of sustainability cannot simply be “affixed” to a project.

Rather, they must be taken into consideration during the entire project cycle, i.e. also

during the planning, preparation and implementation or operation of the project, for

example when reviewing the ability of the project to adjust to exogenous shocks.

3.4. Considering the aspect of sustainability in final evaluations by KfW

(1) In KfW’s final project evaluations sustainability is not treated as a separate

evaluation criterion because an assessment of a project's developmental success

must also consider the future (remaining) useful life or duration of the project, taking

account of any and all remaining risks, and it must include references to the risks.

Instead, the aspect of sustainability plays a role in the evaluation of the key criteria of

effectiveness, relevance/significance and efficiency: in other words, KfW’s final

evaluations assess the sustainable effectiveness, the sustainable

relevance/significance and the sustainable efficiency of a project. This is

illustrated by the box below:

- 14 -

Evaluation level: Evaluation criterion: Relevant questions: Input

↓

Sustainable efficiency

Is the project output generated / used / marketed with sufficient consideration of microeconomic and macroeconomic requirements during the relevant project period?

Output

↓

Sustainable effectiveness

Are the targets that the project is meant to fulfil being achieved to a sufficient degree through specific production and supply levels during the relevant project period?

Outcome

↓

Sustainable significance / relevance

Are the overall objectives that the project is supposed to meet being achieved to a sufficient degree through the use of the generated goods and services by final consumers/the target group as well as through the impacts generated in this connection during the relevant project period?

Impact

Only such a comprehensive viewpoint can prevent the impact assessments from

being limited to what is left of a project after a critical point in time (for example,

whether the road is still passable year-round) instead of determining what the project

has brought about: both results are equally important.

(2) Taking sustainability into account when assessing the effectiveness,

significance/relevance and efficiency raises considerable practical problems that

can differ substantially for each sector and country - resulting in a complicated

bundle of sustainability criteria. The FC project portfolio comprises a heterogeneous

bundle of different project types for which common evaluation criteria can only be

determined extremely broadly.

(3) Since these criteria are not fulfilled in equal measure, separate classifications

are to be made in accordance with the six-step scale of performance evaluation for

the three key criteria of effectiveness, significance/relevance and efficiency. The

three partial classifications are then to be combined to a final classification;

depending on the project, it is entirely possible that the partial classifications are

weighted differently. Such differences in the weighting must be explained, however.

- 15 -

(4) In addition, varying sector-specific minimum requirements may be defined for all

three key criteria, the non-fulfilment of which unavoidably leads to an overall

evaluation of “not successful.“ In case of a negative rating for

significance/relevance or effectiveness (i.e. inadequate achievement of the project and/or the overall objectives) a positive evaluation is clearly not possible.

The same applies for a lack of (micro)economic viability of the project-executing agency as well as for the occurrence of serious undesired side effects (e.g. in the

area of the environment [owing to values that exceed or do not satisfy value limits],

poverty or gender) that are classified as non-compensable in development-policy

terms. Although such additional requirements are frequently not mentioned explicitly

when the overall objectives and project purposes are laid down, their non-fulfilment

is, to a certain degree, a knock-out criterion for project success.

A lack of efficiency of investments in real capital can, ultimately, only be

compensated in exceptional cases; see also Annex 2.

3.5. Projects with limited sustainability requirements

(1) In the transition countries in southeastern Europe and in the former Soviet Union

in particular but also in other regions (such as Central Africa), projects are sponsored

that differ from the usual principles of promotion owing to the special circumstances

in the country in question. This mainly involves the two types of projects described

below:

a) Project type 1 covers refugee and emergency aid measures. In most cases

they aim to help the people affected out of their situation of acute need for a

limited period of time. A structural solution for the emergency situation – e.g.

through a positive development of the overall sectoral conditions – is, however,

not expected. The focus is on temporary project effects, the specific form of

which then determines the classification.

b) Project type 2 covers measures with capacity-building or capacity-enhancing effects that do not, however, meet the standard criteria for

sustainability (e.g. use over the entire technical lifetime) owing to difficult

development-policy conditions (e.g. implementation of a transformation process

- 16 -

that still has to be carried out over a long period of time). Rather, cutbacks in the sustainable project effectiveness are tolerated.

In concert with the BMZ, lower expectations of the level of sustainability are already

accepted during the project appraisal for both project types owing to the adverse

overall conditions – a level that is below the usual minimum criteria for the sector.

Accordingly, during the final evaluation as well the success of such projects must be

relative, i.e. in proportion to the special circumstances. For this projects that were

implemented under comparable circumstances with a similar orientation may be

used for comparison.

(2) In the case of project type 1, which is less common in FC, the performance

rating can still be outlined relatively easily because of clearly set goals: This project

type can already be considered as “successful“ (rating 3 or better) if the intended

solution for the situation of acute need was found quickly and has an adequate

scope, regardless of whether the development of the overall (sectoral) conditions has

been positive.

(3) Project type 2 is considerably more difficult. Ultimately, it reflects a conflict

between, on the one hand, the fact that the performance rating cannot be measured

in terms of minimum sectoral requirements, the non-fulfilment of which was already

taken into account in the decision on a project’s eligibility for financial support. On the

other hand, a project result that is poor compared with other projects should not be

classified as “very successful“ simply because the deficiencies had been

“anticipated.“ For this reason, KfW applies the following principle when it evaluates

individual cases:

a) The evaluation “relatively successful“ (performance rating category 3) is

admissible if the development of the (sectoral) conditions suggests that the

capacities that have been generated will be used to an acceptable degree in the

future as well. In such a case the facilities must be operated over part of their

technical useful life for a period that is still acceptable (which, in turn, must be

defined for each project separately) and used properly despite the fact that the

minimum sector-policy requirements were not achieved and the sustainability of

the facilities is limited (both predicted early on in the project appraisal).

b) The classification of such projects into categories 2 and 1 is possible when

additional structural effects are targeted or achieved. The fulfilment of the

- 17 -

following two preconditions is usually required for classification into

performance rating category 2:

I) Despite the limited target level for sustainability, specific steps defined

during the project appraisal for the structural improvement of the overall

conditions were introduced during the project;

II) at the time of the final evaluation it can be plausibly shown that the

country, sector or project-executing agency will continue on this path to

reform, i.e. that the initial reform steps will not be reversed in the ensuing

period. In countries with many years of exactly the opposite FC

experience, such evidence of plausibility will probably be more difficult to

furnish than in reform-oriented transition countries, to which a certain

amount of confidence can be assigned at the outset, or in post-conflict

situations.

c) Classification into performance rating category 1 should be limited to projects

under which especially convincing steps to overcome the defined obstacles to

sustainability and the achievement of minimum sector requirements have been

taken, and in connection with which the implementation of further measures on

the same developmental path can already be clearly observed.

(4) In the end, however, only evaluation on a case-by-case basis is possible since

most “special circumstances“ are of a unique nature.

4. The Informative Value of Final Evaluations

(1) Whether an FC project is/was sustainable cannot be determined until the planned

project lifetime has ended. All interim statements may refer only to the impacts during

the project's past (certain statements) and to the impacts expected in the future

(projections, uncertain statements). Accordingly, all statements concerning

sustainability refer to a certain point in time and may be invalidated at short notice by

diverse external factors that are frequently unpredictable and cannot be influenced by

the responsible project-executing agency. In fact, it is difficult to make solid

predictions of the various risk factors in the relatively difficult context of developing

countries over long projection periods. In KfW’s final evaluations, though, the

institution’s many years of operational experience definitely provide a solid

foundation for making realistic estimates of the risks in the remaining years of

- 18 -

operation (residual risks). High residual risks normally lead to a project classification

of “not successful.” This also matches the experience KfW has accumulated over

many years.

(2) Thus, the partner’s ability to react to factors that influence the project’s internal

and external sustainability is important. It is becoming increasingly essential to

assess the prerequisites for sustainable project impacts without losing sight of the

aspects of practicability and labour economics. In principle KfW already tries to take

all those factors into consideration that contribute to sustainability in the individual

sectors to varying degrees during project preparation and throughout the entire

project support period until the final evaluation and - if necessary and possible - to

influence them according to certain goals. For this reason as well KfW is placing

greater emphasis on issues that have been proven to be particularly relevant for

sustainable project success: more intense scrutiny of project-executing agencies,

greater consideration of issues relating to operation, participation of the target group

already in the planning phase, possibilities of continuous monitoring of impacts

during project support by KfW, etc.

(3) The informative value of final evaluations in terms of development policy is limited

in that in principle, only the developmental success of projects that have undergone

an final evaluation can be determined, but not the impact of German FC. The reason

is the release or diversion effect that can be observed under FC, which is a result

of the fungibility of the funds. This effect is given, even if it is frequently limited.

Therefore, every FC evaluation that is restricted to projects and does not take the

overall economic or social context into consideration would be quite narrow. Thus, at

KfW the general economic and social context is already included in the project

appraisal. The final evaluations examine the manner in which this was carried out in

order to uncover possible exogenous causes of insufficient project sustainability.

(4) Long-term effects beyond the end of the project can only develop if the

corresponding environment (e.g. overall positive economic development or a

corresponding political priority assigned to the sector) promotes this or even makes it

possible at all. This comment is coupled by the empirical finding that structural impacts that really are long-lasting can only be achieved in a sector if a project is

embedded in larger programmes operated jointly with other donors or the

government of the partner country so that synergy effects can be produced, or if the

project in question is distinctly exemplary.

- 19 -

(5) Ex-post evaluations are not suited for making corrections in ongoing projects

through control measures. Instead, this requires monitoring of impacts. Yet, ex-

post evaluations can teach us what should be taken into more consideration in new

projects than was the case in the past. Accordingly, final evaluations should

determine what lessons KfW can learn out of its experience with the project that can

be applied to the implementation of similar projects. These include, among others,

experience with cooperation with other donors and also experience that is useful for

the further development of sectoral or regional promotional concepts or the ”practical

test” of these concepts.

(6) On various occasions the value of ex-post evaluations is questioned by citing the

fact that most off-target developments were already identified during the project

period and that counteractive measures have already been applied. In consequence,

ex-post evaluations hardly reveal any new, relevant information. This viewpoint

probably holds true with regard to the effectiveness of most exogenous conditions. In

contrast, the influence of numerous determinants of sustainability endogenous to

the project cannot be determined until the external support has come to an end. The

assumption that it is quite clear how things will continue once the support ends – i.e.

‘things have been fine up to now and therefore this positive development will

continue in the future as well’ – is easily proven wrong. As shown by experience with

final evaluations, the opposite of what was expected once the financial support

ended has often been the case, a development that could not be anticipated.

Sometimes, after the support ends energy of the partner is released that was unable

to unfold before; in other cases externally induced, unforeseen problems arise during

the course of the project that have to be resolved. Therefore, KfW regards the final

evaluation procedure as necessary for it to accomplish its tasks, but not for broad-

scale application. The purpose would be better met by concentrating the final

evaluations on projects in which there is particular cognitive interest, or that serve as

a representative random test of the project portfolio pending evaluation. Distinct

selection criteria would have to serve as the basis for identifying such projects.

However, that is not a topic of this paper. Furthermore, current policy on final

evaluations could still be expanded by adding impact-oriented (interim) evaluations of

ongoing priority area programmes in order to eliminate weaknesses already identified

during the term of the projects and thus to open up the opportunity for better results.

- 20 -

Annex 1: KfW’s performance rating categories for project evaluation

a) Successful projects

Rating 1: Very high or high degree of developmental success

According to pertinent evaluation criteria, the project meets all requirements to a high

or very high degree. The evaluation is positive without restriction. The project is

suited to serve as an example of “best practice.”

Rating 2: Satisfactory degree of developmental success

The project meets all pertinent criteria satisfactorily, either constantly or it has minor

weaknesses in some areas that are compensated fully by extraordinarily positive

effects in other areas.

Rating 3: Overall sufficient degree of developmental success

The project achieves a sufficient degree of developmental success overall, either

consistently or on the basis of the total scope of the individual effects. Thus, there

may be major deficiencies in some areas as long as they are compensated by

higher-than-average positive effects in other areas. There may not be deficiencies in

any area that are so serious that they unavoidably lead to an unfavourable overall

evaluation (ratings 4-6) despite all other project effects.

b) Developmental failures

Rating 4: Overall slightly insufficient degree of developmental success

Overall the project does not meet (remains just below) the minimum requirements for

positive effectiveness. This could result from a consistent, minimal shortfall or from

insufficient compensation of major deficiencies in certain areas.

Rating 5: Clearly insufficient degree of developmental success

The minimum requirements are not met by a wide margin but this does not mean that

the best solution would be to terminate the project or discontinue its operation

instead of continuing to use the capacities that have been created.

Rating 6: The project is a total failure

For the most part, the project is useless, or the negative effects are so serious or

outweigh the positive effects to such an extent that the project has either already

been terminated/its operation has already been discontinued or such a step is

necessary due to its uselessness or in order to limit damages.

- 21 -

Annex 2: Individual aspects of project classification6

In practice, the main criteria are seldom met consistently well, or they are

inadequately met. In some cases, this involves a conflict of interests, and the

necessity of a case-by-case evaluation arises.

Whereas, on the one hand, classification into categories 1 and 6 is usually simple

because the situation is clear and the regularity of thoroughly favourable or

unfavourable evaluations according to all relevant criteria is easy to recognize, the

question of compensation for insufficient effects in some areas is most

important for classification into categories 3 and 4. In borderline cases, there are

frequently combinations of insufficient economic efficiency coupled with particularly

high developmental relevance and significance (e.g. project reaches out to the poor),

and vice versa. Here, only the absolute (within the meaning of non-compensable)

minimum requirements of a project's efficiency can be laid down.

The following economic rates of return7 - as real yields on investments (with a long-

term capital commitment) that can be achieved without any major remaining risks -

are defined as absolute minimum requirements for classification into category 3:

a) for projects in very poor developing countries8 with extremely limited potential for development • normal case: 3% p.a. ; • if the distribution/poverty effects are significantly positive and there are no

weaknesses in other areas that require mention: 0% 9 b) for projects in the other developing countries

• normal case: 6% p. a. • exceptional case (see above): 3% p.a.

This is a pragmatic definition applicable to all cases in which no differentiating

assessments (e.g. of the opportunity costs of capital that are specific to the country)

that are similar to other evaluation standards are available.

In case countries in category a) do not give rise to the assumption that even the

minimum requirements laid down for the relevant group regarding the overall

6 Source of Annex 2: KfW Handbook, Annex 3.06. 7 The economic rate of return can naturally only be applied as a minimum standard for projects for which a

cost-benefit analysis can likewise be applied as an instrument for evaluation. 8 For the benefit of a pragmatic and, at the same time, clear use of the term, it covers all countries recognized

by the United Nations as LDCs and which generally receive financial contributions under FC. 9 This lower limit for projects with highly positive distribution/poverty effects is only applicable if the

accepted difference from the overall economic profitability of the individual recipient (3% or 6%, respectively) that would otherwise be accepted as the minimum amount remains within acceptable limits and the project’s capital intensity can be considered appropriate.

- 22 -

economic productivity/maintenance of capital can be attained owing to serious policy

deficiencies, this estimate should be forwarded to the BMZ together with the country

plans for future cooperation. Under certain circumstances, one possibility may be to

agree on requirements to be applied in case the BMZ decides to continue the

cooperation under the aspect of survival aid/world social aid or for other, primarily

political reasons. These lesser requirements agreed and laid down in concert with the

BMZ, which may require the cancellation of the degree of development originally

expected, can then be applied in final evaluations in relevant countries. Yet, capital

losses are only acceptable to a certain degree, despite survival aid and other

assistance with a lower target level. In these cases special emphasis must be placed

on deciding whether the financing of consumer goods should take priority over

investments involving long-term capital commitment.

Since the calculation of the economic rate of return (e.g. Cost Benefit Analysis) is

only possible for a small number of project types, in the remaining cases the

corresponding sector-specific operational evaluation criteria that were defined must

be applied, such as cost coverage, maximum unit costs or minimal capacity

utilization. Thus, under the aspect of efficiency an average capacity utilization over

the lifetime of the investment of less than two-thirds of the nominal capacity will

normally not lead to classification as a successful project.

From a financial perspective, under the aspect of sustainability assurance of

liquidity over the lifetime of the investment (for the individual project or business unit)

must be applied as a key minimum requirement. In this context coverage of the costs

of current operation and of the spare parts acquired at short intervals through service

fees or user tariffs or fees is basically indispensable. However, this minimum

requirement may be fulfilled on the overall company level (cross-subsidies), but it

must generally be backed up by a careful analysis of the economic situation of the

project-executing agency. Shortages in cost coverage always involve deductions in

terms of both the sustainability of the solution to the problem and of the allocation

efficiency. Therefore, they have to be justified.

Exceptions from the minimum requirements mentioned above for liquidity

safeguarding as well as cost coverage via service fees are admissible for projects

with high socio-political importance or with other crucial developmental effects

(including significant, positive external effects). This refers mainly to the sectors of

education, health care and family planning, but also to infrastructure facilities for

- 23 -

which service fees may not be charged (e.g. dikes), or tend to be too complicated in

most cases (e.g. roads). Nevertheless, one minimum requirement that is

indispensable in these cases as well is securing financing for operation and

maintenance, which must then come from other sources (general budget allocations,

fees and taxes levied especially for this purpose, etc).

As regards classification into category 3 (or better) the non-compensable deficiencies of a project include environmental impacts still considered intolerable

as well as unacceptable social and socio-cultural impacts.