Post on 28-Mar-2015
transcript
Experiences concerning fuzzy-verification and pattern recognition methods
Ulrich Damrath
U. Damrath: Experiences concerning fuzzy-verification ... - COSMO GM, Offenbach 2009
2
Outlook
• Results on operational verifcation for winter and summer month
• An approach concerning significance test of „fuzzy“-verification results
• Estimation of consistency of forecasts using a pattern recognition method (CRA method by Beth Ebert)
U. Damrath: Experiences concerning fuzzy-verification ... - COSMO GM, Offenbach 2009
3
Fractions skill score for forecasts of GME, COSMO-EU and COSMO-DE for December 2008, forecast time 06-18 hours
GME COSMO-EU
COSMO-DE
U. Damrath: Experiences concerning fuzzy-verification ... - COSMO GM, Offenbach 2009
4
ETS upscaling for forecasts of GME, COSMO-EU and COSMO-DE for December 2008, forecast time 06-18 hours
GME COSMO-EU
COSMO-DE
U. Damrath: Experiences concerning fuzzy-verification ... - COSMO GM, Offenbach 2009
5
Fractions skill score for forecasts of GME, COSMO-EU and COSMO-DE for August 2009, forecast time 06-18 hours
Global Europe
Germany
U. Damrath: Experiences concerning fuzzy-verification ... - COSMO GM, Offenbach 2009
6
ETS upscaling for forecasts of GME, COSMO-EU and COSMO-DE for August 2009, forecast time 06-18 hours
Global Europe
Germany
U. Damrath: Experiences concerning fuzzy-verification ... - COSMO GM, Offenbach 2009
7
Examination of statistical significance of „fuzzy“-verification results using bootstrapping
• Basic idea of bootstrapping:• Repeat a resampling all elements of a given in a sample of
forecasts and observations as often as necessary (N times) and calculate the relevant score(s)
• Calculate from N scores statistical properties of the sample such as mean value standard deviation, confidence intervals and quantiles
• Application to „fuzzy“-verification• Resampling is done using „blocks“.
• Blocks are defined as single days.
• Number of resampling cases: N=Days*100
• Calculation scores from N samples for NT thesholds and NW windows
• Calculation of quantiles for each window and threshold
U. Damrath: Experiences concerning fuzzy-verification ... - COSMO GM, Offenbach 2009
8
Values and quantiles 0.1 and 0.9 for Upscaling ETS GME, period June - August 2009
Germany
U. Damrath: Experiences concerning fuzzy-verification ... - COSMO GM, Offenbach 2009
9
Values and quantiles 0.1 and 0.9 for Upscaling ETS COSMO-EU, period June - August 2009
Germany
U. Damrath: Experiences concerning fuzzy-verification ... - COSMO GM, Offenbach 2009
10
Values and quantiles 0.1 and 0.9 for Upscaling ETS COSMO-DE, period June -August 2009
Germany
U. Damrath: Experiences concerning fuzzy-verification ... - COSMO GM, Offenbach 2009
11
Next step: Evaluation of significance
• First impression: Is the result of Model 1 better than the result of Model 2?
• Significance hypothesis checked using a Wilcoxon-test (IDL-code RS_TEST)
U. Damrath: Experiences concerning fuzzy-verification ... - COSMO GM, Offenbach 2009
12
Differences between GME and COSMO-EU
ETS(COSMO-EU) - ETS(GME) Significance test
COSMO-EU better than GMECOSMO-EU worse than GME
Germany
U. Damrath: Experiences concerning fuzzy-verification ... - COSMO GM, Offenbach 2009
13
Differences between GME and COSMO-DE
ETS(COSMO-DE) - ETS(GME) Significance test
COSMO-DE better than GMECOSMO-DE worse than GME
Germany
U. Damrath: Experiences concerning fuzzy-verification ... - COSMO GM, Offenbach 2009
14
Differences between COSMO-DE and COSMO-EU
ETS(COSMO-DE) - ETS(COSMO-EU) Significance test
COSMO-DE better than COSMO-EUCOSMO-DE worse than COSMO-EU
Germany
U. Damrath: Experiences concerning fuzzy-verification ... - COSMO GM, Offenbach 2009
15
ETS(COSMO-DE) - ETS(COSMO-EU)
COSMO-DE better than COSMO-EUCOSMO-DE worse than COSMO-EU
Significance test
Differences between COSMO-DE and COSMO-EU
U. Damrath: Experiences concerning fuzzy-verification ... - COSMO GM, Offenbach 2009
16
Example of good precipitation forecast of COSMO-DE
Zeigten die numerischen Modelle und die statistischen Prognose----------------------------------------------------------------verfahren Signale für das Ereignis?-----------------------------------Die Numerik zeigte im Vorfeld vermehrt Signale für kräftige Konvektion. Während diese bei GME und COSMO-EU recht breit gestreut und pauschal auftraten, signalisierten mehrere COSMO-DE-Läufe eine linienartige Struktur mit unwetterartigen Zellen (auf Basis der 1- bzw. 3-stündigen RR-Prognosen) im Grenzbereich von Hessen zu NRW und Niedersachsen. Diese Linie trat dann in den Mittags- und frühen Nachmittagsstunden tatsächlich auf, wenn auch nicht 100%ig kongruent, aber doch in der Nähe, so dass in diesem Fall von einer guten Prognose gesprochen werden kann (mehr dazu siehe "Zentraler UW-Sofortbericht" der VBZ).
U. Damrath: Experiences concerning fuzzy-verification ... - COSMO GM, Offenbach 2009
17
Example of good precipitation forecast of COSMO-DE
3h-precipitation forecast of COSMO-DEvalid 10.08.2009 12 UTC, left 03 UTC +9h, right 06 UTC +6h.
3h-precipitation observation10.08.2009 09-12 UTC
U. Damrath: Experiences concerning fuzzy-verification ... - COSMO GM, Offenbach 2009
18
Example of good precipitation forecast of COSMO-DE compared to other models
U. Damrath: Experiences concerning fuzzy-verification ... - COSMO GM, Offenbach 2009
19
Example of good precipitation forecast of COSMO-DE compared to other models
U. Damrath: Experiences concerning fuzzy-verification ... - COSMO GM, Offenbach 2009
20
About consistency and inconsistency
• Forecasters are interested in consistent model forecasts.• But due to growing of errors during forecast time forecasts
consistency cannot be expected concerning all properties of the forecasted fields!
• Inconsistency: Differences between forecasts that are valid for the same time concerning different properties of the forecasted fields (properties of the pattern, values at special points of interest, extreme values, ...)
• Differences between the forecasted fields concerning• phase,• amplitude• and the remaining part
Entity-based QPF verification (rain “blobs”) by E. Ebert (BOM Melbourne)Verify the properties of the forecast rain system against the properties of
the observed rain system:
• location• rain area• rain intensity
(mean, maximum)
Observed Forecast
CRA error decompositionThe total mean squared error (MSE) can be written as:
MSEtotal = MSEdisplacement + MSEvolume + MSEpattern
Configuration for the current study:
- “Observations”: forecasts: 06-30 hours
- Forecasts : forecasts: 30-54 hours and
forecasts: 54-78 hours
U. Damrath: Experiences concerning fuzzy-verification ... - COSMO GM, Offenbach 2009
22
Dark :forecasts 30-54 hLight:forecasts 54-78 h
U. Damrath: Experiences concerning fuzzy-verification ... - COSMO GM, Offenbach 2009
23
Dark :forecasts 30-54 hLight:forecasts 54-78 h
U. Damrath: Experiences concerning fuzzy-verification ... - COSMO GM, Offenbach 2009
24
Dark :forecasts 30-54 hLight:forecasts 54-78 h
U. Damrath: Experiences concerning fuzzy-verification ... - COSMO GM, Offenbach 2009
25
Dark :forecasts 30-54 hLight:forecasts 54-78 h
U. Damrath: Experiences concerning fuzzy-verification ... - COSMO GM, Offenbach 2009
26
Summary• Scores like Fractions skill score and ETS from upscaling show in general
advantages of COSMO models compared to GME.• This is true especially for summer months.
• For winter months all models have nearly the same quality for low precipitation amounts and large window sizes for averaging.
• Significance test lead to the results, that:• The advantages of COSMO models compared to GME are statistically significant for
most window sizes and precipitation amounts.
• The differences between COSMO-EU and COSMO-DE are not significant altough there are systematical differences for different precipitation amounts and window sizes.
• There are some cases with very useful precipitation forecasts of COSMO-DE compared to COSMO-EU from the view of forecasters.
• A study about the consistency of precipitation forecasts showed - it could be expected , but now it is proved - that:
• Forecasts of high precipitation amounts are less consistent than those for low precipitation amounts.
• Pattern errors contribute most to forecast errors.
• During winter months volume errors are higher than displacement errors.
• During summer months displacement errors are higher than volume errors.