VERIFICATIONHighligths
by WG5
2
Outlook
• Some focus on Temperature with common plots and Conditional Verification
• Some Fuzzy verification
• Long trends
SON 2009
DJF 2009-2010
5 Verification results at MeteoSwiss in 2010
COSMO GM / WG5 Parallel Session, 06.09.2010
T2m: mean diurnal cycle (first 24h forecasts)domain Switzerland (hourly SYNOP‘s)
Autumn 2009
Winter 2009/2010
Spring 2010
P. Kaufmann, V. Stauch
OBS
COSMO-7
COSMO-2
Summer 2010
T2m COSMO-I7 00UTC: LAST YEAR
WAM
WG5 COSMO General Meeting, Moscow 2010
Conditional VerificationExtracting information for relevant
performance of weather parameters
The input from modelers and forecasters is necessary for identifying and testing
hypotheses.
F. Gofa - HNMS
Temp in overcast conditionsTemp in overcast conditions
Fall Winter
Spring Summer
F. Gofa - HNMS
COSMO General Meeting – Moscow 06-10 Sept 2010
Conditional Verification Temp – TCC obs >=75%
SON
MAM
DJF
Better behaviour for all the seasonsCompare to no condition model
Temp in clear sky conditions
Fall Winter
Spring Summer
F. Gofa - HNMS
Conditional Verification Temp – TCC obs <=35%
Worse behaviour for all the seasonsCompare to no condition model
SON
MAM
DJF
Temp in ‘calm’ conditions (<2 m/s)
Fall Winter
Spring Summer
WG5 COSMO General Meeting, Moscow 2010F. Gofa - HNMS
WG5 COSMO General Meeting, Moscow 2010
Fall Winter
Spring Summer
Temp in ‘high wind’ conditions >10m/s
F. Gofa - HNMS
14
Some conclusion
• A problem with Temp is clear.
• RMSE between 2-3 °C it is not so small.
• Diurnal cycle too cold during the day and too warm during the night
• Clear different behaviour with conditions on TCC and with different wind conditions
15
Outlook
• Some focus on Temperature with common plots and Conditional Verification
• Some Fuzzy verification
• Long trends
16 Verification results at MeteoSwiss in 2010
COSMO GM / WG5 Parallel Session, 06.09.2010
results for 20093h accumulated precipitation sumsover the domain of the swiss radar composit
models: COSMO-2 and COSMO-7leadtimes 04 – 07h for all 8 daily forecast runs
obervation precipitation estimates of the swiss radar composit
in case of a missing value, the full date will not be evaluated(total of 28 days)
Neighborhood verification for precipitation
17 Verification results at MeteoSwiss in 2010
COSMO GM / WG5 Parallel Session, 06.09.2010
COSMO-2 COSMO-7 COSMO-2 - COSMO-7
- =
- =
goodbadCOSMO-7 better COSMO-2 better
Fractions Skill Score
Upscaling
Neighborhood (fuzzy) verification 2009, FSS and UP T. Weusthoff
18 Verification results at MeteoSwiss in 2010
COSMO GM / WG5 Parallel Session, 06.09.2010
FractionsSkill Score
FSS
UpscalingETS
Upscalingfreq. bias
FBI
Neighborhood (fuzzy) verification: Spring 2010COSMO-2/COSMO-7: 3h acc, leadtime +4 to +6 for all models
COSMO-2 COSMO-7 IFS
T. Weusthoff
19
Outlook
• Some Common Plots (Task 6 Versus)
• Conditional Verification
• Some Fuzzy verification
• Long trends verification
Introduction of new Z0
Introduction of new Z0
Introduction of new Z0
Introduction of new Z0
Total cloud cover_____ Cloud cover above 2 Octa (Cl.1).......... Cloud cover above 6 Octa (Cl.2)
Valid time 00 UTC
Cloud cover of low clouds because incorporation of AWS
Stand Mai 2010
Time series of the COSI: State May 2010
Stand Mai 2010
Time series of the COSI: State May 2010
Time series of the COSI: Temperature day 1
Stand Mai 2010
Time series of the COSI: Temperature day 2
Stand Mai 2010
Time series of the COSI: Temperature day 3
Stand Mai 2010
Time series of the COSI: State May 2010 (STDV used for T2m instead of RMSE)
• Long period verification (seasonal trend) (from djf’04 to mam’10)
1. Some Statistical indices for low thres (0.2mm/24h)
2. Some Statistical indices for high thres (20mm/24h)
• Verification ovest last year (DJF 2009-MAM2010)
1. Driving model comparison:
ecmwf/Cosmo-I7/Cosmo-I2
2. Driving model comparison:
ecmwf/Cosmo-ME/Cosmo-IT
COSMO General Meeting – Moscow 06-10 Sept 2010
• All the versions present a seasonal cycle with an overestimation during summertime (except COSMO-7 and I2)• COSMO-7 and I2 underestimate• Overestimation error decreases in D+2 (spin-up effect vanished)
QPF verification of the 4 model versions at 7 km res. (COSMO-I7, COSMO-7, COSMO-EU, COSMO-ME) with the 2 model versions at 2.8 km res. (COSMO-I2, COSMO-IT)Dataset: high resolution network of rain gauges coming from COSMO dataset and Civil Protection Department 1300 stationsMethod: 24h/6h averaged cumulated precipitation value over 90 meteo-hydrological basins
Seasonal trend - low thresholds
COSMO General Meeting – Moscow 06-10 Sept 2010
• Very light improvement in trend
• Seasonal error cycle: lower ets during winter and summertime
• no significant differences between D+1 and D+2
• Last winter (very snowy particularly in Northern Italy): low ets value (D+1 and D+2) model error or lack of representativeness of the rain gauges over the plain during snowfall ?
Seasonal trend - low thresholds
COSMO General Meeting – Moscow 06-10 Sept 2010
Driving model comparison: ECMWF/COSMO-ME/COSMO-IT, low thresholds
• ECMWF tendency to forecast low rainfall amounts big overestimation, big false alarms, very low ets, quite good pod
• Better prediction for COSMO-models (no strong differences between ME and IT)
• Seasons DJF2009 – MAM2010
COSMO General Meeting – Moscow 06-10 Sept 2010
• ECMWF tendency to forecast low rainfall amounts big overestimation, big false alarms, very low ets, quite good pod
• Better prediction for COSMO-models BUT bad performance during summertime
•Seasons DJF2009 – MAM2010
Driving model comparison: ECMWF/COSMO-I7/COSMO-I2, low thresholds
37 Verification results at MeteoSwiss in 2010
COSMO GM / WG5 Parallel Session, 06.09.2010
Precipitation (12h-sums +12 to +24h):Spring 2010 over Switzerland (SYNOP‘s)COSMO-7 & COSMO-2
V. Stauch
38 Verification results at MeteoSwiss in 2010
COSMO GM / WG5 Parallel Session, 06.09.2010
Precipitation (12h-sums +12 to +24h):Spring 2010 over Switzerland (SYNOP‘s)COSMO-7 & IFS
V. Stauch
COSMO General Meeting – Moscow 06-10 Sept 2010
• Slight bias reduction during latest seasons
• Last winter: all the versions overestimate (probably due to lack of representativeness of the rain gauges over the plain during snowfall)
• Strong COSMO-7 underestimation BUT slight improvement during latest seasons
Seasonal trend - high thresholds
COSMO General Meeting – Moscow 06-10 Sept 2010
• Low values during summertime
• In general, quite stationary error since son2008 up to now
• All the versions present a jump around son2008: ets increases from 0.2-0.4 up to 0.3-0.5
• Skill decreases with forecast time
Seasonal trend - high thresholds
COSMO General Meeting – Moscow 06-10 Sept 2010
• ECMWF difficulty to forecast high rainfall amounts bias around 1 BUT big false alarms, very low ets and pod
• Better prediction for COSMO-models
• Seasons DJF2009 – MAM2010
Driving model comparison: ECMWF/COSMO-ME/COSMO-IT, high thresholds
COSMO General Meeting – Moscow 06-10 Sept 2010
• ECMWF difficulty to forecast high rainfall amounts bias around 1 BUT big false alarms, very low ets and pod
• Better prediction for COSMO-models
• Seasons DJF2009 – MAM2010
Driving model comparison: ECMWF/COSMO-I7/COSMO-I2, high thresholds
12h Precipitation – Sep2009-Aug2010
WG5 COSMO General Meeting, Moscow 2010
COSMOECMWF
Really strong overestimation of lower preci amounts up to 3mm and lower ETS scores for
ECMWF
F. Gofa - HNMS
44
Some conclusion
• Long term trends show a general (sometimes light) improvements for all the considered models
• Comparison between COSMO models and IFS shows a general clear better behaviour for COSMO implementations