ECMWF forecast User Meeting -- Reading, June 2006 Slide 1
Verification of weather parameters
Anna Ghelli, ECMWF
ECMWF forecast User Meeting -- Reading, June 2006 Slide 2
Overview
Deterministic forecast performance for different weather parameters
Precipitation forecast: scores and their confidence
SYNOP on the GTSPrecipitation analysis
Ensemble prediction System: its performance relative to precipitation and 10m wind
ECMWF forecast User Meeting -- Reading, June 2006 Slide 3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
de
g C
-0.2
-0.1
0
0.1
0.2
0.3
0.4
J
1999A J O J
2000A J O J
2001A J O J
2002A J O J
2003A J O J
2004A J O J
2005A J O J
2006
fc error of indate findate wp step ntot BIAS STDV RMSE SKILL MAE SKILL CORR M-OB M-FC
skil 48h skil 54h skil 60h skil 66h
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
de
g C
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
J
1999A J O J
2000A J O J
2001A J O J
2002A J O J
2003A J O J
2004A J O J
2005A J O J
2006
fc error of indate findate wp step ntot BIAS STDV RMSE SKILL MAE SKILL CORR M-OB M-FC
skil 48h skil 54h skil 60h skil 66h
North America
Europe
2m TemperatureSkill (rmse) for different forecast ranges
Top panel: North America
Bottom panel: Europe
Strong seasonality in the skill with higher values during winters.
2mT skill have reached a plateau. The skills for different timesteps have levelled off. The two past winters were dominated by prolonged periods of cold weather.
ECMWF forecast User Meeting -- Reading, June 2006 Slide 4
1.8
1.9
2
2.1
2.2
2.3
2.4
2.5
2.6
2.7
2.8
m /
s
1.8
1.9
2
2.1
2.2
2.3
2.4
2.5
2.6
2.7
2.8
J
1999A J O J
2000A J O J
2001A J O J
2002A J O J
2003A J O J
2004A J O J
2005A J O J
2006
fc error of indate findate wp step ntot BIAS STDV RMSE SKILL MAE SKILL CORR M-OB M-FC
rmse 48h rmse 54h rmse 60h rmse 66h
-0.05
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
g /
kg
-0.05
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
J
1999A J O J
2000A J O J
2001A J O J
2002A J O J
2003A J O J
2004A J O J
2005A J O J
2006
fc error of indate findate wp step ntot BIAS STDV RMSE SKILL MAE SKILL CORR M-OB M-FC
skil 48h skil 54h skil 60h skil 66hPerformance for different forecast ranges Europe
Europe
Top panel: Specific humidity
Bottom panel: 10m wind speed
Higher skill (MAE) in winter. Last four winters have consistently kept higher level of
performance
RMSE: Changes in the forecasting model have improved the performance of the model in forecasting wind speed.
ECMWF forecast User Meeting -- Reading, June 2006 Slide 5
a b
c d
Observed yes Observed no
Forecast
yes
Forecast
no
1. FREQUENCY BIAS INDEX
caba
FBI
2. TRUE SKILL SCORE
3. HIT RATE
3. FALSE ALARM RATE
db
b
ca
aTSS
db
bF
ECMWF forecast User Meeting -- Reading, June 2006 Slide 6
0
0.2
0.4
0.6
TS
S
0
0.2
0.4
0.6
1992M
1993J S DM
1994J S DM
1995J S DM
1996J S DM
1997J S DM
1998J S DM
1999J S DM
2000J S DM
2001J S DM
2002J S DM
2003J S DM
2004J S DM
2005J S DM
2006
precipitation exceeding 15.0 mm/24h
t + 42 t + 90
15mm/24hEurope
24 hour accumulated precipitation verified against SYNOP on GTS
The forecast is reduced to a yes/no event by selecting thresholds. Confidence intervals have been plotted for each TSS value.
High thresholds have large confidence intervals, important to remember when assessing performance of the system
t+42
t+90
ECMWF forecast User Meeting -- Reading, June 2006 Slide 7
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
FB
I
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
S1993
DM1994
J S DM1995
J S DM1996
J S DM1997
J S DM1998
J S DM1999
J S DM2000
J S DM2001
J S DM2002
J S DM2003
J S DM2004
J S DM2005
J S DM2006
precipitation exceeding 1.0 mm/24h
t + 42 t + 90Europe24 hour accumulated precipitation verified against SYNOP on GTS
The forecasting system over-estimate the number of events for thresholds of 1 mm/24h. A decrease of FBI was observed when in the autumn 1999, when vertical resolution was increased and a new convection scheme was implemented. Further changes in the convection during 2003 have slightly increased FBI values
FBI measures the ratio between the frequency of the forecast events and the frequency of the observed events. FBI>1 over-estimateFBI<1 under-estimate
t+42: solid shadingt+90: dotted shading
ECMWF forecast User Meeting -- Reading, June 2006 Slide 8
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
FB
I
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
S1993
DM1994
J S DM1995
J S DM1996
J S DM1997
J S DM1998
J S DM1999
J S DM2000
J S DM2001
J S DM2002
J S DM2003
J S DM2004
J S DM2005
J S DM2006
precipitation exceeding 15.0 mm/24h
t + 42 t + 90
FBI decreases to values closer to 1 as we increase the threshold, but higher thresholds have larger confidence intervals!
24 hour accumulated precipitation verified against SYNOP on GTS
Europe 15mm/24h t+42: solid shadingt+90: dotted shading
ECMWF forecast User Meeting -- Reading, June 2006 Slide 9
0
0.2
0.4
0.6
TSS
0
0.2
0.4
0.6
1993M
1994M
1995M
1996M
1997M
1998M
1999M
2000M
2001M
2002M
2003M
2004M
2005 2006
precipitation exceeding 15.0 mm/24h
t + 42 t + 90
0.2
0.4
0.6
TSS
0.2
0.4
0.6
1993D
1994D
1995D
1996D
1997D
1998D
1999D
2000D
2001D
2002D
2003D
2004 2005
precipitation exceeding 15.0 mm/24h
t + 42 t + 90
15mm/24h
DJF
MAM
Green: t+42Light blue: t+90
15mm/24h
Stratifying the sample: selecting seasons as rule
ECMWF forecast User Meeting -- Reading, June 2006 Slide 10
40°N
50°N
60°N
70°N
40°W
40°W
20°W
20°W
0°
0°
20°E
20°E 40°E
40°E
1
2
5
10
15
20
25
40
40°N
50°N
60°N
70°N40°W
40°W
20°W
20°W
0°
0°
20°E
20°E 40°E
40°E
60°E
60°E
1
2
5
10
15
20
25
40
Precipitation analysis for Europe
•High density networks in Europe (Member and Co-operating states)•Upscaling (simple box averaging to obtain a areal precipitation value)
ECMWF forecast User Meeting -- Reading, June 2006 Slide 11
5
10
1515
20
50°N
60°N
0°
0°
1
2
5
10
15
20
25
40
Each grid box will contain a certain number of stations. The number of stations will not be constant every day.
The number of stations per grid box indicates how representative the analysis is for the specific grid point.
ECMWF forecast User Meeting -- Reading, June 2006 Slide 12
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
TS
S
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
J2002
S D M2003
J S D M2004
J S D M2005
J S
precipitation exceeding 1.0 mm/24h
t + 42 t + 42
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
TS
S
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
J2002
S D M2003
J S D M2004
J S D M2005
J S
precipitation exceeding 1.0 mm/24h
t + 90 t + 90
Green : proxy
t+42
t+90
1 mm/24h
Which reference system?
Use proxy: the shortest possible forecast verifying at 6UTC. We assume the proxy to be the best forecast. Its TSS values (verification against precipitation analysis) will give us an upper limit for the verification of other forecast ranges.
ECMWF forecast User Meeting -- Reading, June 2006 Slide 13
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
TS
S
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
J2002
S D M2003
J S D M2004
J S D M2005
J S
precipitation exceeding 15.0 mm/24h
t + 90 t + 90
15 mm/24h
t+90
Which reference system?
Use proxy: the shortest possible forecast verifying at 6UTC. We assume the proxy to be the best forecast. Its TSS values (verification against precipitation analysis) will give us an upper limit for the verification of other forecast ranges.
Green : proxy
ECMWF forecast User Meeting -- Reading, June 2006 Slide 14
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1.1
1.2
1.3
1.4
1.5
1.6
1.7
1.8
1.9
2
FB
I
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1.1
1.2
1.3
1.4
1.5
1.6
1.7
1.8
1.9
2
J2002
S D M2003
J S D M2004
J S D M2005
J S
precipitation exceeding 15.0 mm/24h
t + 66 t + 66 t + 66 t + 66
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1.1
1.2
1.3
1.4
1.5
1.6
1.7
1.8
1.9
2
FB
I
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1.1
1.2
1.3
1.4
1.5
1.6
1.7
1.8
1.9
2
J2002
S D M2003
J S D M2004
J S D M2005
J S
precipitation exceeding 1.0 mm/24h
t + 66 t + 66 t + 66 t + 66
Europe
FBI plotted for two thresholds (1mm/24h, and 15mm/24h)•Verification against SYNOP on the GTS (greenish shading), •Verification against precipitation analysis (blue dotted)
FBI values are higher (lower) in the verification against SYNOP on the GTS (analysis) for lower (higher) thresholds. Forecast range t+66
ECMWF forecast User Meeting -- Reading, June 2006 Slide 15
Europe
TSS (threshold 25mm/24h) plotted for t+66•Verification against SYNOP on the GTS (greenish shading), •Verification against precipitation analysis (blue dotted)
TSS values decrease as we increase forecast range.
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
TS
S
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
J2002
S D M2003
J S D M2004
J S D M2005
J S
precipitation exceeding 25.0 mm/24h
t + 66 t + 66 t + 66 t + 66
ECMWF forecast User Meeting -- Reading, June 2006 Slide 16
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1994A
1995A D A
1996A D A
1997A D A
1998A D A
1999A D A
2000A D A
2001A D A
2002A D A
2003A D A
2004A D A
2005A D
2006
Brier skill score (sample clim) fc step 96 24h-precipitation exceedingProbability forecastverification against obs ( 3-M. moving sample)
1 mm 5 mm 10 mm 20 mm
Timeseries of Brier Skill Score for Europe
The BSS is written as 1- BS/BSref Sample climate is the reference system
BS measures the mean squared difference between forecast and observation in probability space. Equivalent to MSE for deterministic forecast
Forecast vs. observations
Improvements back in Autumn 1999 – High thresholds performance down at the beginning of 2005 probably linked to drier conditions over Europe. Dec-Apr 2006 performance better than previous 3 years.
Increased resolutionT255 T399
ECMWF forecast User Meeting -- Reading, June 2006 Slide 17
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
OCT
2000FEB
2001JUN OCT FEB
2002JUN OCT FEB
2003JUN OCT FEB
2004JUN OCT FEB
2005JUN OCT
2006
Brier skill score (sample clim) fc step 96 24h-precipitation exceedingProbability forecastverification against obs ( 3-M. moving sample)
1 mm 5 mm 10 mm 20 mm
Forecast vs. observations
ECMWF forecast User Meeting -- Reading, June 2006 Slide 18
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
A
1996D A
1997A D A
1998A D A
1999A D A
2000A D A
2001A D A
2002A D A
2003A D A
2004A D A
2005A D
2006
Brier skill score (long term clim) fc step 96 10m wind speed exceedingProbability forecastverification against an ( 3-M. moving sample)
10 m/s 15 m/s
Timeseries of Brier Skill Score for Europe
The BSS is written as 1- BS/BSref Sample climate is the reference system
BS measures the mean squared difference between forecast and observation in probability space. Equivalent to MSE for deterministic forecast C
Forecast vs analysis
Increased resolutionT255 T399
10 m Wind speed
ECMWF forecast User Meeting -- Reading, June 2006 Slide 19
0.2
0.2
0.4
0.4
0.6
0.6
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
ob
s fr
equ
ency
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
forecast probability
67515918
3400
2853
2555
2301
2269
2322
2443
2777
5345
sample clim = 0.26 BS = 0.157 SSBS = 0.19 bias(ctr) = 0.2820041001-20050430 STEP 96 24-hour precipitation gt 1.0
0
0.1
0 0.2 0.4 0.6 0.8 1
rel FC distribution
sample clim
0.2
0.2
0.4
0.4
0.6
0.6
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
ob
s f
req
ue
nc
y
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
forecast probability
167569578
3778
2385
1665
1263
1047
841
643
519
459
sample clim = 0.11 BS = 0.079 SSBS = 0.21 bias(ctr) = 0.2820041001-20050430 STEP 96 24-hour precipitation gt 5.0
0
0.1
0.2
0.3
0.4
0.5
0 0.2 0.4 0.6 0.8 1
rel FC distribution
sample clim
EuropeRainy season (24h accum.): October to April Forecast range: t+96Verification against SYNOP on GTS
2005-20061mm/24h BS=0.079
2004-20051mm/24h BS=0.157
Consistent picture for the two seasons
0.2
0.2
0.4
0.4
0.6
0.6
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
ob
s fr
equ
ency
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
forecast probability
70746333
3457
2786
2431
2243
2100
2014
2163
2388
4970
sample clim = 0.13 BS = 0.071 SSBS = 0.39 bias(ctr) = 0.1020051001-20060501 STEP 96 24-hour precipitation gt 1.0
0
0.1
0 0.2 0.4 0.6 0.8 1
rel FC distribution
sample clim
0.2
0.2
0.4
0.4
0.6
0.6
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
ob
s f
req
ue
nc
y
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
forecast probability
172069045
3300
2092
1566
1206
982
784
627
541
610
sample clim = 0.06 BS = 0.036 SSBS = 0.33 bias(ctr) = 0.1020051001-20060501 STEP 96 24-hour precipitation gt 5.0
0
0.1
0.2
0 0.2 0.4 0.6 0.8 1
rel FC distribution
sample clim
ECMWF forecast User Meeting -- Reading, June 2006 Slide 20
0.2
0.2
0.4
0.4
0.6
0.6
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
ob
s fr
equ
ency
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
forecast probability
259208057
2096
1070
649410
268
183137
7965
sample clim = 0.05 BS = 0.040 SSBS = 0.16 bias(ctr) = 0.2820041001-20050430 STEP 96 24-hour precipitation gt 10.0
0
0.2
0.4
0.6
0.8
0 0.2 0.4 0.6 0.8 1
rel FC distribution
sample clim
0.2
0.2
0.4
0.4
0.6
0.6
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
ob
s f
req
ue
nc
y
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
forecast probability
346083414
477199
95
60
33
2220
4
2
sample clim = 0.01 BS = 0.012 SSBS = 0.12 bias(ctr) = 0.2820041001-20050430 STEP 96 24-hour precipitation gt 20.0
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
rel FC distribution
sample clim
EuropeRainy season (24h accum.): October to AprilForecast range: t+96Verification against SYNOP on GTS
2004-200510 mm/24h BS=0.04
2005-200610 mm/24h BS=0.019
Consistent picture for the two seasons. Higher thresholds better reliability
0.2
0.2
0.4
0.4
0.6
0.6
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
ob
s fr
equ
ency
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
forecast probability
255427522
1984
1007
612
423296
224
153
100
96
sample clim = 0.03 BS = 0.019 SSBS = 0.25 bias(ctr) = 0.1020051001-20060501 STEP 96 24-hour precipitation gt 10.0
0
0.1
0.2
0.3
0.4
0 0.2 0.4 0.6 0.8 1
rel FC distribution
sample clim
0.2
0.2
0.4
0.4
0.6
0.6
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
ob
s f
req
ue
nc
y
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
forecast probability
336453303
532
214
116
60
37
24
10
13
5
sample clim = 0.01 BS = 0.007 SSBS = 0.12 bias(ctr) = 0.1020051001-20060501 STEP 96 24-hour precipitation gt 20.0
0
0.1
0.2
0.3
0.4
0.5
0 0.2 0.4 0.6 0.8 1
rel FC distribution
sample clim
ECMWF forecast User Meeting -- Reading, June 2006 Slide 21
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
hit
rat
e
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
false alarm rate
20041001-20050430 24-hour precipitation gt 1.0
0
2000
4000
6000
8000
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
STEP 96
0
2000
4000
6000
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
STEP 120
t+ 96 A=0.864
t+120 A=0.830
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
hit
ra
te
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
false alarm rate
20041001-20050430 24-hour precipitation gt 5.0
0
5000
0.100 105
0.150 105
0.200 105
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
STEP 96
0
5000
0.100 105
0.150 105
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
STEP 120
t+ 96 A=0.864
t+120 A=0.832
EuropeRainy season (24h accum.): October to AprilForecast range: t+96Verification against SYNOP on GTS
2004-20055 mm/24h
2005-20065 mm/24h
Consistent picture for the two seasons.
Full symbol: T511/799Shape: T255/T399
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
hit
rat
e
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
false alarm rate
20051001-20060501 24-hour precipitation gt 1.0
0
2000
4000
6000
8000
0.100 105
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
STEP 72
0
2000
4000
6000
8000
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
STEP 96
0
2000
4000
6000
8000
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
STEP 120
t+ 72 A=0.899
t+ 96 A=0.877
t+120 A=0.849
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
hit
ra
te
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
false alarm rate
20051001-20060501 24-hour precipitation gt 5.0
0
5000
0.100 105
0.150 105
0.200 105
0.250 105
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
STEP 72
0
5000
0.100 105
0.150 105
0.200 105
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
STEP 96
0
5000
0.100 105
0.150 105
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
STEP 120
t+ 72 A=0.909
t+ 96 A=0.889
t+120 A=0.860
t+96 : A=0.864
t+120: A=0.832
t+96 : A=0.889
t+120: A=0.860
ECMWF forecast User Meeting -- Reading, June 2006 Slide 22
0.5
0.55
0.6
0.65
0.7
0.75
0.8
0.85
0.9
0.95
1
RO
CA
0.5
0.55
0.6
0.65
0.7
0.75
0.8
0.85
0.9
0.95
1
J1996
S D M1997
J S D M1998
J S D M1999
J S D M2000
J S D M2001
J S D M2002
J S D M2003
J S D M2004
J S D M2005
J S D M2006
24 hour total precipitation verified against observations t+ 96
threshold = 1 threshold = 5 threshold = 10 threshold = 20
Increased resolution
EuropeROC Area Verification against SYNOP on GTS for t+96
T255 T399
ECMWF forecast User Meeting -- Reading, June 2006 Slide 23
Conclusion 2m Temperature: the skill over
Europe has reached a plateau. Over
N.America winters are more skilful
than summers.
Specific humidity shows consistent
skills. Winters more skilful than
summers
Wind: Recent changes in the model
have resulted in a decrease of the
RMSE over Europe.
TCC: New cloud scheme was
introduced in April 2005. Forecasts
of certain cloud types have
improved.
Importance of confidence intervals
Precipitation forecast
improvements are slow, but evident.
FBI indicates over-estimation of
small threshold events
verification against precipitation
analysis shows a better picture.
Precipitation analysis can be used
for verification in a delayed mode. It
can be used to define the
performance of a reference system.
ECMWF forecast User Meeting -- Reading, June 2006 Slide 24
Brier skill score and ROC area: In
recent year the system has
maintained its good performance.
The increase in horizontal
resolution has had a small but
positive impact on both
precipitation and wind speed.
Reliability diagrams for the last two
rainy seasons show a skilful
system, with an evident decrease of
Brier Score for the 2005-2006
season.
Conclusion