1

Adjusting Development Outcomes by Risk

Risk-Adjusted Expected Development Outcome of Private Sector Investment Projects

IEG Evaluation WeekOctober 24, 2011

Hiroyuki HatashimaIEG Private Sector Evaluation

Outline of the presentation

2

• Performance measurement vs risk• Drivers of development outcome for

IFC investment projects• Model based on drivers• Two applications of the model

‒ Performance assessment‒ Expected development outcomes

Performance measurement vs risk

3

• Risk is measurable possibilities of losing or not gaining value

• Risk-return trade-off as the basis of investment management(if you don’t want risk, don’t expect return)

• For fair comparison, the return has to be compared to the risk undertaken.

Private Sector Development Outcome

Development Outcome = Overall effect on country’s development

BusinessPerformance

Financiers

Economic Performance

Gov’t, taxpayersemployees, customers,

suppliers

Envi. & SocialPerformance

Neighbors & Environment

Private SectorDevelopment

Demo. effects, linkages, corp. gov, investment climate

& new entrants

4

Model based on Development Outcome Results Drivers

External to IFC -Risks

Sponsor Risk

Market Risk

Changes in Country

Business Climate

Project Type

Internal to IFC – Work

Quality

Screening, Appraisal & Structuring

Supervision & Administration

Role & Contribution

DO = f ( )+ +

5

Risk factors influencing development outcomes

Risk Factors DescriptionCoeff.

against DO

Signif. at 5%

Sponsor RiskExperience, financial capacity, commitment and reputation of sponsors (1=high risk, 0=low risk)

-0.21 Yes

Market RiskBusiness competitiveness in the market, distortions (1=high risk, 0=low risk)

-0.14 Yes

Changes in Country Business Climate

Changes in Country Risk indicators between approval and evaluation (i.e. 5 years) (score @ eval – score @ approval)

0.009 Yes

Project Type “greenfield” (high risk) vs. expansion project (1=greenfield, 0=expansion) -0.06 No

Based on 2000-2010 XPSR Evaluation, N=655Coefficient against DO in multiple variable regression. R2=0.09 6

Regression results of all drivers

7

Factors Coefficient Signif. at 5%Sponsor Risk -0.11 Yes

Market Risk -0.12 Yes

Changes in Country Business Climate 0.005 Yes

Project type -0.07 No

Screening, appraisal & structuring work quality

0.33 Yes

Supervision & administration work quality 0.28 Yes

IFC role and contribution 0.46 Yes

Based on 2000-2010 XPSR Evaluation, N=655Coef. for multiple variable regression. R2=0.37

Two applications of the model

8

1. Performance assessment• Outcomes with focus on IFC Work

Quality, taking risks

2. Likely outcomes• Based on existing risks and work

quality scenarios

Use of model 1: Performance Assessment

• Focusing on IFC controllable factors, given risks taken• Which is the best performing region, adjusted by risk,

focusing on WQ improvement?Region Development Outcome

high %

Southern Europe & Central Asia 87%

Middle East & North Africa 80%

Latin America & Caribbean 79%

Sub-Saharan Africa 74%

South Asia 72%

East Asia & Pacific 62%

Central & Eastern Europe 60%

IFC overall 73% 9

Two Work Quality scenarios for comparison

Scenarios: Potential BenchmarkWQ assumptions

Perfect WQ = 100% High WQ across the board

WQ ratings of previous periods

Risks Actual project risks

Indicator means

Model-generated maximum possible DO rating

expected success rate expected to outperform, adjusted by the embedded project risk factors.

Comparison with actual outcome

Indicate gaps to potential based on existing risks

Reflecting WQ changes

DO= f (Risks + Work Quality) +

10

Gaps between actual and potential/benchmark = risk adjusted outcome indicators

Region Actual Potential BenchmarkSouthern Europe & Central Asia

87% 90% 85%

Middle East & North Africa

80% 86% 59%

Latin America & Caribbean

79% 92% 83%

Sub-Saharan Africa

74% 88% 56%

South Asia 72% 90% 62%

East Asia & Pacific 62% 89% 55%

Central & Eastern Europe

60% 90% 83%

IFC overall 73% 89% 72%11

-35 -30 -25 -20 -15 -10 -5 0

-30

-20

-10

0

10

20

30

IFC

MENASS AFR

S AsiaEAP

LAC

CEE

SECA

(Potential – Actual)

(Ben

chm

ark-

Act

ual)

MENA & AFR outcomes stood out, adjusted by risks/Work Quality.

12

-30 -25 -20 -15 -10 -5 0 5

-15

-10

-5

0

5

10

15

MAS AFR/LAC

INF EMENA

FM EMENA

FM AFR/LACIFC

FM Asia

INF AFR/LAC

MAS EMENA

MAS Asia

INF Asia

INFRA (AFR/LAC/EMENA) and FM (AFR/LAC) outcomes stood out

(Ben

chm

ark-

Act

ual)

(Potential – Actual)

13

Use of model 2: likely outcomes

1996

-98

1997

-99

1998

-00

1999

-01

2000

-02

2001

-03

2002

-04

2003

-05

2004

-06

2005

-07

2006

-08

2007

-09

2008

-10

0%10%20%30%40%50%60%70%80%90%

XPSR Development Outcome

Non IDAIDA

Evaluation Year

DO h

igh

%

IDA – Non IDA gap in recent years: will it continue?

14

What’s behind the gap?

• Project Risks– IDA more riskier than non IDA

1995-97

1996-98

1997-99

1998-00

1999-01

2000-02

2001-03

2002-04

2003-05

0%

10%

20%

30%

40%

50%

60%

70%

Sponsor Risk

Approval Year

High

Ris

k %

1995-97

1996-98

1997-99

1998-00

1999-01

2000-02

2001-03

2002-04

2003-05

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

Market Risk

NonIDAIDA

Approval Year

High

Ris

k %

15

What’s behind the gap?• Changes in Country Business Climate

- IDA lower than non IDA

1987

1989

1991

1993

1995

1997

1999

2001

2003

2005

2007

2009

-10

-5

0

5

10

15

20

IDAnon-IDA

Evaluation year

Chan

ges

in II

CCR

s co

res

16

What’s behind the gap?Appraisal WQ and Roles dropped in IDA

1996

-98

1997

-99

1998

-00

1999

-01

2000

-02

2001

-03

2002

-04

2003

-05

2004

-06

2005

-07

2006

-08

2007

-09

2008

-100%

20%

40%

60%

80%

Appraisal Work Quality

Evaluation Year

1996

-98

1997

-99

1998

-00

1999

-01

2000

-02

2001

-03

2002

-04

2003

-05

2004

-06

2005

-07

2006

-08

2007

-09

2008

-100%

20%

40%

60%

80%

100%

Supervision Work Quality

Evaluation Year

1996

-98

1997

-99

1998

-00

1999

-01

2000

-02

2001

-03

2002

-04

2003

-05

2004

-06

2005

-07

2006

-08

2007

-09

2008

-100%

20%

40%

60%

80%

100%

Role and Contribution

Non IDAIDA

17

Prospects for coming years

•Assume 08-10 XPSR WQ (IDA average)• Actual risk of 2006-09 approvals (except

country risk – changes so far) – what will be the likely results?

1995

-97

1996

-98

1997

-99

1998

-00

1999

-01

2000

-02

2001

-03

2002

-04

2003

-05

2004

-06

2005

-07

2006

-08

2007

-090%

10%20%30%40%50%60%70%

Sponsor Risk

Approval Year

High

Ris

k %

1995

-97

1997

-99

1999

-01

2001

-03

2003

-05

2005

-07

2007

-090%

20%

40%

60%

80%

Market Risk

NonIDAIDA

Approval Year

18

Changes in country business climate so far

1987

1988

1989

1990

1991

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

2011

*20

12*20

13*20

14*20

15*

-10

-5

0

5

10

15

20

Changes in IICCR Score between approval and evalua-tion

(or approval and most recent score for 2006-2010 ap-provals = 2011-15 maturity)

IDAnon-IDA

Evaluation Year (approval + 5 years)

Chan

ges

in II

CCR

scor

e

•IDA – Not dropping sharply as non-IDA

19

Future prospects: IDA-non IDA gap will narrow

• If above WQ assumptions hold

1996-98

1997-99

1998-00

1999-01

2000-02

2001-03

2002-04

2003-05

2004-06

2005-07

2006-08

2007-09

2008-10

2009-11

2010-12

2011-13

2012-14

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

Development Outcome (incl. projection)

Non IDAIDA

Evaluation year

Deve

lopm

ent O

utco

me

High

%

= Predicted by the model 20

Summary

• Project performance = need to separate factors controlled by institution vs risks

• Multi-factor Model can generate success rates that

= can treat as benchmarks for performance measurement;

= can indicate future outcomes, given risks and work quality assumptions

21

22

Adjusting Development Outcomes by Risk

Risk-Adjusted Expected Development Outcome of Private Sector Investment Projects

IEG Evaluation WeekOctober 24, 2011

Hiroyuki HatashimaIEG Private Sector Evaluation