Evaluating Density Forecasts with Applications to ESPF · Testing i.i.d. normal x t x t 1 t H 0 :...

transcript

Evaluating Density Forecasts with Applications to ESPFKanemi Ban (Osaka University)Masaaki Kawagoe (ESRI)Hideaki Matsuoka (JCER)

ESRI-JCER Conference

Feb. 21, 2013

1. Introduction

2. Methodology

3. Data

4. Results

5. Conclusion

1. Introduction ESPF’s consensus forecasts are of great practical use,

… … and its quality is confirmed by annual performance

reviews.

Record of best 5

0 1 2 3 4 5 total

number of forecasters

30 10 4 3 2 1 50

Records of being selected as “best 5 forecasters”

CF is here!

1-2 Introduction

“Risk” information is valuable to ESPF users. Introduced density forecasts survey in June

2008. ⇒ “Mean Probability Distribution” (MPD) A natural next questions to be addressed is:

“How good is it and in what sense?”

1. Introduction

2. Methodology

3. Data

4. Results

5. Conclusion

2-1 Methodology

Two approaches to density forecastsPIT (Probability Integral Transform)Scoring the densities

PIT: Diebold, Gunther and Tay (1998) the sequence of PIT of realized variable with

respect to the observed density forecast is i.i.d. U(0,1), if the observed density forecast coincides with the sequence of conditional density of a target variable.

2-2 Diebold, Gunther and Tay (1998)

ty ,...,, 321 tttt yyy

ttt yf |

ttt yp |

Series of realizations

Information set available at t

Conditional density of a series

Corresponding sequence of forecast density

mtttt yfyp 11 || to be tested

is not observed and may exhibit structural change.

ttt yf |

If , then is .

tttt yPduupz

2-3 Probability Integral Transform

tttttt

tttt zPp

zPfzPfz

zPzq 1

Density of tz

ttt yfyp tt zq 1,0U

tttt zPyand

zzPzp 1

coincide with .

are generated from .

2-4 i.i.d. uniform distribution

mtty 1

mttttt yf 1|

mttttt yf 1| mttttt yp 1|

11 Uduupz diimy

111111

|...|||,...,,

yfyfyfyyyf

mmmmmm

|...||

|,...,,

zPpzPf

2-5. Berkowitz (2001) How to test i.i.d. U(0,1) DGT (1998) advocated graphical or nonparametric

approaches. But those methods are data intensive. Unfortunately, ESPF has only 4 year data. Berkowitz (2001) proposed to use a simple

transformation to normality. Likelihood ratio test is available and more powerful in small samples.

ttt duufzx 11 1,0... Ndii

Standard normal cdf.

tt x Standard normal distribution function

tttttt

xfxxh ^

loglog ^

tt xh Density of based on tx upt

loglog ^ tttttttt xxhxfxf ^

Testing i.i.d. normal

ttt xx 1

2.7 Scoring rules RPS (Ranked Probability Score)Boero, Smith and Wallis (2011)Kenny, Kostka, and Masera (2012)

A density forecast of individual j: Bin: CDF of : CDF of realization : , which is binary.

Deviation values are used for aggregation12

)( itjt Bp

itB ),...1( Ki

)( itjt Bp )( itjt BP

1 1, )(1

itittjj YBP

131313

1. Introduction

2. Methodology

3. Data

4. Results

5. Conclusion

3-1-1 MPDs for real GDP growth rates

‐8-7-6-5-4-3-2-101234

806 809 812 903

2008 GDP Density Forecast by Periods

-8-7-6-5-4-3-2-101234

806 809 812 903 906 909 912

-7-6-5-4-3-2-1012345

906 909 912 1003 1006 1009 1012 1103

-8-7-6-5-4-3-2-101234

1006 1009 1012 1103 1106 1109 1112 1203

151515

3-1-2 MPDs for CPI Inflation Rates

-4-3-2-101234

806 809 812 903

2008 CPI Density Forecast by Periods

-4-3-2-101234

806 809 812 903 906 909 912 1003

-4-3-2-101234

906 909 912 1003 1006 1009 1012 1103

-4-3-2-101234

1006 1009 1012 1103 1106 1109 1112 1203

3-2-1 Fan Charts for real GDP growth rates

Note: The prediction intervals covers 10%, 25%,50%,75% and 90% ranges.

3-2-2 Fan Charts for CPI inflation rates

Note: The prediction intervals covers 10%, 25%,50%,75% and 90% ranges.

-2-1.5

-1-0.5

2005 2006 2007 2008 2009

June 2008 CPI Forecast

-2-1.5

-1-0.5

2005 2006 2007 2008 2009 2010

-2-1.5

-1-0.5

2005 2006 2007 2008 2009 2010 2011

-2-1.5

-1-0.5

2005 2006 2007 2008 2009 2010 2011

How many bins are used?

1 2 3 4 5 6 7 8 9 10 11 12 13 14

GDP CPI

(%)＜FY2008＞

1 2 3 4 5 6 7 8 9 10 11 12 13 14

GDP CPI

(%)＜FY2009＞

1 2 3 4 5 6 7 8 9 10 11 12 13 14

GDP CPI

(%) ＜FY2010＞

101520253035

1 2 3 4 5 6 7 8 9 10 11 12 13 14

GDP CPI

(%) ＜FY2011＞

GDP CPIFY2008 FY2009 FY2010 FY2011 FY2008 FY2009 FY2010 FY2011

Average 3.88 4.66 4.35 4.59 Average 3.89 4.51 4.53 4.53STD 1.34 1.98 1.74 1.79 STD 1.32 1.64 1.78 1.66# 428 631 677 679 # 346 551 599 623

Are realized values outliers? : probability of being contained in density forecasts

0102030405060708090

6 7 8 9 10 11 12 1 2 3 4 5

GDP CPI

＜FY2008＞

Year t Year t+1

0102030405060708090

1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5

GDP CPI

＜FY2009＞

Year t Year t+1

0102030405060708090

1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5

GDP CPI

＜FY2010＞

Year t Year t+1

0102030405060708090

1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5

GDP CPI

＜FY2011＞

Year t Year t+1

202020

1. Introduction

2. Methodology

3. Data

4. Results

5. Conclusion

1. Probability integral transform with respect to the forecast densities

2. If , then those data are dropped. (“Failure”)

3. transform of

4. Test of in the autoregressive model

5. is rejected, the same procedure is undertaken by subgrouping individual forecasters.

4-1-1 Procedures to test independency

titi duupz ,,

10 ,, titi zorz

titi zx ,1

tititi xx ,1,,

4-1-2 Samples to test independency

period # of sample # of failure

Real GDP growth rate

June 2008 36 36June 2009 27 20June 2010 37 1June 2011 40 1

CPI inflation rateJune 2008 34 3June 2009 30 6June 2010 37 7June 2011 40 22

National account FY (April to March) figures are available in mid-May: the June forecast is the first with non-overlapping information.

Sample size: 34 for GDP in FY2010 and 2011; 29 for CPI in FY2008 to FY2011.

4-1-3 Results of the independency test

Eq.1 Eq.2 Eq.3Dependent var. GDP growth rate CPI inflation rateρ estimate 0.055 -0.587 -0.605

standard error 0.255 0.236 0.565p-value 0.830 0.013 0.285

Sample period FY 2010 to 2011 FY 2008 to 2011 FY 2008 to 2011# of observations 34 29 14# of individuals 20 10# of instruments 4 4

tititi xx ,1,, Estimate equations: The null hypothesis:

Rejected in Eq.2, but not rejected in Eq.3 for 10 individuals with better forecast performance

4-2-1 Procedures to calculate RPS Want to examine performance of MPD. Use of June samples is justified by

independency test results, but the sample is small.

4 cases: (June or 17 months) x (exclude FY2008 or not)Larger samples: 17 months for GDP and 16

months for CPI. Compare MPD with three benchmarks and

individuals’ density forecasts.24

4-2-2 Benchmark density forecasts: Real GDP Growth Rate in June 2011

4-2-3 Benchmark density forecasts: CPI Inflation in June 2011

4-2-4 Results of RPS calculations: comparing with benchmarks

period evaluation MPD Uniform Normal NaïveJune samples (1)

GDP FY2009 to 2011 RPS 1.29 1.34 1.56 1.37 ADV of RPS 46.26 56.47 57.00 48.14

including FY2008 RPS 3.10 2.81 3.15 n.a. ADV of RPS 46.22 47.08 51.18 n.a.

CPI FY2009 to 2011 RPS 0.45 1.06 0.39 0.51ADV of RPS 46.35 57.86 45.06 47.68

including FY2008 RPS 0.42 1.01 0.33 n.a. ADV of RPS 46.19 57.58 44.35 n.a. 17 or 16 month samples for each FY (2)

GDP FY2009 to 2011 ADV of RPS 46.78 55.34 51.36 48.94 including FY2008 ADV of RPS 46.80 52.87 49.98 51.27

CPI FY2009 to 2011 ADV of RPS 46.06 61.08 44.36 48.48including FY2008 ADV of RPS 46.13 59.24 45.14 48.88

Note: (1) Calculated from individuals without missing responses. (2) Calculated from individuals with more than 70 per cent responses.

4-2-5 Results of RPS calculations: comparing with individuals (performance ranking)

period MPD Uniform Normal Naïve # of individuals

June samples (1)

GDP FY2009 to 2011 7 31 31 16 33including FY2008 2 3 23 n.a. 29

CPI FY2009 to 2011 13 28 9 17 32including FY2008 12 30 2 n.a. 32

17 or 16 month samples for each FY (2)

GDP FY2009 to 2011 5 36 32 13 37including FY2008 3 34 19 31 36

CPI FY2009 to 2011 6 34 3 15 35including FY2008 4 34 3 17 35

Note: (1) Calculated from individuals without missing responses. (2) Calculated from individuals with more than 70 per cent responses.

292929

1. Introduction

2. Methodology

3. Data

4. Results

5. Conclusion

5-1 Conclusion Apply Berkowitz’s (2001) test to individuals’

density forecasts produced in June every year. Real GDP growth rates Fail to reject the independency in real in FY 2010

and 2011. CPI inflation rates Reject the independency in all the samples in FY

2008 and 2011.But fail to reject it in its subsample with better

forecast performance.

5-2 Conclusion

Calculate RPSMPD is a “good” density forecast compared to

three benchmark as well as individual densities.The above is robust to changes in sample periods

and variables.Prudent CPI density?Overconfidence may get rewarded in GDP.

ReferencesBerkowitz, J. (2001) “Testing Density Forecasts, With Applications to Risk

Management”, Journal of Business and Economic Statistics 19:465-474.Boero, Gianna, Jeremy Smith, and Kenneth F. Wallis (2011) “Scoring Rules

and Survey Density Forecasts,” International Journal of Forecasting 27: 379–393

Diebold, F.X., T.A. Gunther and A.S. Tay (1998) “Evaluating Density Forecasts with Applications to Financial Risk Management”, International Economic Reviews 39:863-883.

Kenny, Geoff, Thomas Kostka, and Federico Masera (2012) “How Informative Are The Subjective Density Forecasts of Macroeconomists,” ECB Working Paper Series, No.1446.

Evaluating Density Forecasts with Applications to ESPF · Testing i.i.d. normal x t x t 1 t H 0 :...

Documents