32
1 Evaluating Density Forecasts with Applications to ESPF Kanemi Ban (Osaka University) Masaaki Kawagoe (ESRI) Hideaki Matsuoka (JCER) ESRI-JCER Conference Feb. 21, 2013
111
Evaluating Density Forecasts with Applications to ESPFKanemi Ban (Osaka University)Masaaki Kawagoe (ESRI)Hideaki Matsuoka (JCER)
ESRI-JCER Conference
Feb. 21, 2013
222
1. Introduction
2. Methodology
3. Data
4. Results
5. Conclusion
3
1. Introduction ESPF’s consensus forecasts are of great practical use,
… … and its quality is confirmed by annual performance
reviews.
3
Record of best 5
0 1 2 3 4 5 total
number of forecasters
30 10 4 3 2 1 50
Records of being selected as “best 5 forecasters”
CF is here!
4
1-2 Introduction
“Risk” information is valuable to ESPF users. Introduced density forecasts survey in June
2008. ⇒ “Mean Probability Distribution” (MPD) A natural next questions to be addressed is:
“How good is it and in what sense?”
4
555
1. Introduction
2. Methodology
3. Data
4. Results
5. Conclusion
666
2-1 Methodology
Two approaches to density forecastsPIT (Probability Integral Transform)Scoring the densities
PIT: Diebold, Gunther and Tay (1998) the sequence of PIT of realized variable with
respect to the observed density forecast is i.i.d. U(0,1), if the observed density forecast coincides with the sequence of conditional density of a target variable.
77
2-2 Diebold, Gunther and Tay (1998)
ty ,...,, 321 tttt yyy
ttt yf |
ttt yp |
Series of realizations
Information set available at t
Conditional density of a series
Corresponding sequence of forecast density
mtttt
mtttt yfyp 11 || to be tested
is not observed and may exhibit structural change.
ttt yf |
ty
88
If , then is .
ty
tttt yPduupz
2-3 Probability Integral Transform
ttt
tttttt
t
tttt zPp
zPfzPfz
zPzq 1
11
1
Density of tz
ttt yfyp tt zq 1,0U
tttt
tttt zPyand
zzPzp 1
99
coincide with .
are generated from .
2-4 i.i.d. uniform distribution
mtty 1
mttttt yf 1|
mttttt yf 1| mttttt yp 1|
1,0..
11 Uduupz diimy
tmtt
t
111111
11
|...|||,...,,
yfyfyfyyyf
mmmmmm
mm
1
111
111
11
1111
11111
1
111
|...||
|,...,,
zPpzPf
zPpzPf
zPpzPf
zzzq
mmm
mmmm
mmm
mmmm
mm
10
2-5. Berkowitz (2001) How to test i.i.d. U(0,1) DGT (1998) advocated graphical or nonparametric
approaches. But those methods are data intensive. Unfortunately, ESPF has only 4 year data. Berkowitz (2001) proposed to use a simple
transformation to normality. Likelihood ratio test is available and more powerful in small samples.
10
ty
ttt duufzx 11 1,0... Ndii
Standard normal cdf.
1111
tt x Standard normal distribution function
tt
tttttt
xf
xfxxh ^
tt
tt
tt
tt
xxh
xf
xf
loglog ^
tt xh Density of based on tx upt
tt
tt
tt
tt
xxh
xf
xf
loglog ^ tttttttt xxhxfxf ^
Testing i.i.d. normal
ttt xx 1
0:0 H
2.7 Scoring rules RPS (Ranked Probability Score)Boero, Smith and Wallis (2011)Kenny, Kostka, and Masera (2012)
A density forecast of individual j: Bin: CDF of : CDF of realization : , which is binary.
Deviation values are used for aggregation12
)( itjt Bp
itB ),...1( Ki
)( itjt Bp )( itjt BP
tytY
2
1 1, )(1
T
t
K
itittjj YBP
TRPS
131313
1. Introduction
2. Methodology
3. Data
4. Results
5. Conclusion
1414
3-1-1 MPDs for real GDP growth rates
‐8-7-6-5-4-3-2-101234
806 809 812 903
2008 GDP Density Forecast by Periods
-8-7-6-5-4-3-2-101234
806 809 812 903 906 909 912
2009 GDP Density Forecast by Periods
-7-6-5-4-3-2-1012345
906 909 912 1003 1006 1009 1012 1103
2010 GDP Density Forecast by Periods
-8-7-6-5-4-3-2-101234
1006 1009 1012 1103 1106 1109 1112 1203
2011 GDP Density Forecast by Periods
151515
3-1-2 MPDs for CPI Inflation Rates
-4-3-2-101234
806 809 812 903
2008 CPI Density Forecast by Periods
-4-3-2-101234
806 809 812 903 906 909 912 1003
2009 CPI Density Forecast by Periods
-4-3-2-101234
906 909 912 1003 1006 1009 1012 1103
2010 CPI Density Forecast by Periods
-4-3-2-101234
1006 1009 1012 1103 1106 1109 1112 1203
2011 CPI Density Forecast by Periods
1616
3-2-1 Fan Charts for real GDP growth rates
Note: The prediction intervals covers 10%, 25%,50%,75% and 90% ranges.
1717
3-2-2 Fan Charts for CPI inflation rates
Note: The prediction intervals covers 10%, 25%,50%,75% and 90% ranges.
-2-1.5
-1-0.5
00.5
11.5
2
2005 2006 2007 2008 2009
June 2008 CPI Forecast
-2-1.5
-1-0.5
00.5
11.5
2
2005 2006 2007 2008 2009 2010
June 2009 CPI Forecast
-2-1.5
-1-0.5
00.5
11.5
2
2005 2006 2007 2008 2009 2010 2011
June 2010 CPI Forecast
-2-1.5
-1-0.5
00.5
11.5
2
2005 2006 2007 2008 2009 2010 2011
June 2011 CPI Forecast
18
How many bins are used?
0
5
10
15
20
25
30
35
1 2 3 4 5 6 7 8 9 10 11 12 13 14
GDP CPI
(%)＜FY2008＞
0
5
10
15
20
25
30
1 2 3 4 5 6 7 8 9 10 11 12 13 14
GDP CPI
(%)＜FY2009＞
0
5
10
15
20
25
30
1 2 3 4 5 6 7 8 9 10 11 12 13 14
GDP CPI
(%) ＜FY2010＞
05
101520253035
1 2 3 4 5 6 7 8 9 10 11 12 13 14
GDP CPI
(%) ＜FY2011＞
GDP CPIFY2008 FY2009 FY2010 FY2011 FY2008 FY2009 FY2010 FY2011
Average 3.88 4.66 4.35 4.59 Average 3.89 4.51 4.53 4.53STD 1.34 1.98 1.74 1.79 STD 1.32 1.64 1.78 1.66# 428 631 677 679 # 346 551 599 623
19
Are realized values outliers? : probability of being contained in density forecasts
0102030405060708090
100
6 7 8 9 10 11 12 1 2 3 4 5
GDP CPI
＜FY2008＞
Year t Year t+1
(%)
0102030405060708090
100
1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5
GDP CPI
＜FY2009＞
Year t Year t+1
(%)
0102030405060708090
100
1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5
GDP CPI
＜FY2010＞
Year t Year t+1
(%)
0102030405060708090
100
1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5
GDP CPI
＜FY2011＞
Year t Year t+1
(%)
202020
1. Introduction
2. Methodology
3. Data
4. Results
5. Conclusion
2121
1. Probability integral transform with respect to the forecast densities
2. If , then those data are dropped. (“Failure”)
3. transform of
4. Test of in the autoregressive model
5. is rejected, the same procedure is undertaken by subgrouping individual forecasters.
4-1-1 Procedures to test independency
ty
titi duupz ,,
1
10 ,, titi zorz
tiz ,
titi zx ,1
,
0:0 H
tititi xx ,1,,
0:0 H
4-1-2 Samples to test independency
period # of sample # of failure
Real GDP growth rate
June 2008 36 36June 2009 27 20June 2010 37 1June 2011 40 1
CPI inflation rateJune 2008 34 3June 2009 30 6June 2010 37 7June 2011 40 22
22
National account FY (April to March) figures are available in mid-May: the June forecast is the first with non-overlapping information.
Sample size: 34 for GDP in FY2010 and 2011; 29 for CPI in FY2008 to FY2011.
4-1-3 Results of the independency test
23
Eq.1 Eq.2 Eq.3Dependent var. GDP growth rate CPI inflation rateρ estimate 0.055 -0.587 -0.605
standard error 0.255 0.236 0.565p-value 0.830 0.013 0.285
Sample period FY 2010 to 2011 FY 2008 to 2011 FY 2008 to 2011# of observations 34 29 14# of individuals 20 10# of instruments 4 4
tititi xx ,1,, Estimate equations: The null hypothesis:
Rejected in Eq.2, but not rejected in Eq.3 for 10 individuals with better forecast performance
0:0 H
4-2-1 Procedures to calculate RPS Want to examine performance of MPD. Use of June samples is justified by
independency test results, but the sample is small.
4 cases: (June or 17 months) x (exclude FY2008 or not)Larger samples: 17 months for GDP and 16
months for CPI. Compare MPD with three benchmarks and
individuals’ density forecasts.24
25
4-2-2 Benchmark density forecasts: Real GDP Growth Rate in June 2011
26
4-2-3 Benchmark density forecasts: CPI Inflation in June 2011
4-2-4 Results of RPS calculations: comparing with benchmarks
27
period evaluation MPD Uniform Normal NaïveJune samples (1)
GDP FY2009 to 2011 RPS 1.29 1.34 1.56 1.37 ADV of RPS 46.26 56.47 57.00 48.14
including FY2008 RPS 3.10 2.81 3.15 n.a. ADV of RPS 46.22 47.08 51.18 n.a.
CPI FY2009 to 2011 RPS 0.45 1.06 0.39 0.51ADV of RPS 46.35 57.86 45.06 47.68
including FY2008 RPS 0.42 1.01 0.33 n.a. ADV of RPS 46.19 57.58 44.35 n.a. 17 or 16 month samples for each FY (2)
GDP FY2009 to 2011 ADV of RPS 46.78 55.34 51.36 48.94 including FY2008 ADV of RPS 46.80 52.87 49.98 51.27
CPI FY2009 to 2011 ADV of RPS 46.06 61.08 44.36 48.48including FY2008 ADV of RPS 46.13 59.24 45.14 48.88
Note: (1) Calculated from individuals without missing responses. (2) Calculated from individuals with more than 70 per cent responses.
4-2-5 Results of RPS calculations: comparing with individuals (performance ranking)
28
period MPD Uniform Normal Naïve # of individuals
June samples (1)
GDP FY2009 to 2011 7 31 31 16 33including FY2008 2 3 23 n.a. 29
CPI FY2009 to 2011 13 28 9 17 32including FY2008 12 30 2 n.a. 32
17 or 16 month samples for each FY (2)
GDP FY2009 to 2011 5 36 32 13 37including FY2008 3 34 19 31 36
CPI FY2009 to 2011 6 34 3 15 35including FY2008 4 34 3 17 35
Note: (1) Calculated from individuals without missing responses. (2) Calculated from individuals with more than 70 per cent responses.
292929
1. Introduction
2. Methodology
3. Data
4. Results
5. Conclusion
30
5-1 Conclusion Apply Berkowitz’s (2001) test to individuals’
density forecasts produced in June every year. Real GDP growth rates Fail to reject the independency in real in FY 2010
and 2011. CPI inflation rates Reject the independency in all the samples in FY
2008 and 2011.But fail to reject it in its subsample with better
forecast performance.
5-2 Conclusion
Calculate RPSMPD is a “good” density forecast compared to
three benchmark as well as individual densities.The above is robust to changes in sample periods
and variables.Prudent CPI density?Overconfidence may get rewarded in GDP.
3232
ReferencesBerkowitz, J. (2001) “Testing Density Forecasts, With Applications to Risk
Management”, Journal of Business and Economic Statistics 19:465-474.Boero, Gianna, Jeremy Smith, and Kenneth F. Wallis (2011) “Scoring Rules
and Survey Density Forecasts,” International Journal of Forecasting 27: 379–393
Diebold, F.X., T.A. Gunther and A.S. Tay (1998) “Evaluating Density Forecasts with Applications to Financial Risk Management”, International Economic Reviews 39:863-883.
Kenny, Geoff, Thomas Kostka, and Federico Masera (2012) “How Informative Are The Subjective Density Forecasts of Macroeconomists,” ECB Working Paper Series, No.1446.
