13rd verification workshop, ECMWF 2007, Göber

„Detective“ Miss Marple in

„Murder at the Gallop“

m1

Slide 1

m1 Now, this is how we verify the forecasters at DWD: we watch them intensily and when they get it wrong, they'll get a bang over their head. OK, I'm slightly exaggerating and I will explain later what Detective Miss Marple has to do with warning verification.mgoeber, 03/01/2007

23rd verification workshop, ECMWF 2007, Göber

Comparing peaches and apples

On the accuracy of gust warnings issued by forecasters and the accuracy of the model

guidance

Martin Göber

Department Weather ForecastingDeutscher Wetterdienst

Acknowledgements: R. Kirchner, T. Kratzsch, D. Richardson, G. Schweigert, S. Tremmel

33rd verification workshop, ECMWF 2007, Göber

43rd verification workshop, ECMWF 2007, Göber

Heidke skill score

00,10,2

0,30,40,50,60,7

0,80,9

1

near gale gale storm violent storm hurrican force

p

forecaster Local model

53rd verification workshop, ECMWF 2007, Göber

hit rate

0

0,1

0,2

0,3

0,4

0,5

0,6

0,7

0,8

0,9

1

near gale gale storm violent storm hurrican force

p

forecaster Local model

63rd verification workshop, ECMWF 2007, Göber

false alarm ratio

0

0,1

0,2

0,3

0,4

0,5

0,6

0,7

0,8

0,9

1

near gale gale storm violent storm hurrican force

p

forecaster Local model

73rd verification workshop, ECMWF 2007, Göber

Frequency bias

0

1

2

3

4

5

6

7

8

9

10

near gale gale storm violent storm hurrican force

p

forecaster Local model

83rd verification workshop, ECMWF 2007, Göber

relative value for C/L=0,1

-0,4

-0,2

0

0,2

0,4

0,6

0,8

1

near gale gale storm violent storm hurrican

p

forecaster Local model

93rd verification workshop, ECMWF 2007, Göber

relative value for C/L=0,01

-0,4

-0,2

0

0,2

0,4

0,6

0,8

1

near gale gale storm violent storm hurrican

p

forecaster Local model

103rd verification workshop, ECMWF 2007, Göber

Signal Detection Theory

113rd verification workshop, ECMWF 2007, Göber

1 2 3 4 5 6 7 8 9 10all cases

0

5

10

15

20

25

30

35

frequ

ency

indicator

all

all cases

(=CAPE, wind speed, finger prints, ....)

123rd verification workshop, ECMWF 2007, Göber

0 1 2 3 4 5 6 7 8 9event

all cases0

5

10

15

20

25

30

35fre

quen

cy

indicator

eventno eventall cases

133rd verification workshop, ECMWF 2007, Göber

0 1 2 3 4 5 6 7 8 9event

all cases0

5

10

15

20

25

30

35

frequ

ency

indicator

eventno eventall cases

thresholdmisses

False alarms

POD=70%FAR=15%ETS=50%Bias=80%

143rd verification workshop, ECMWF 2007, Göber

0 1 2 3 4 5 6 7 8 9event

all cases0

5

10

15

20

25

30

35

frequ

ency

indicator

eventno eventall cases

thresholdmisses

False alarms

POD=90%FAR=40%ETS=42%

Bias=150%

153rd verification workshop, ECMWF 2007, Göber

„Detective“ Miss Marple in

„Murder at the Gallop“

m2

Slide 15

m2 Now, this is how we verify the forecasters at DWD: we watch them intensily and when they get it wrong, they'll get a bang over their head. OK, I'm slightly exaggerating and I will explain later what Detective Miss Marple has to do with warning verification.mgoeber, 03/01/2007

163rd verification workshop, ECMWF 2007, Göber

1

10

100

1000

10000

1000000 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39

m/s

frequ

ency

n(F|O=29)

Violent storm warning: gusts>= 29 m/s

173rd verification workshop, ECMWF 2007, Göber

1

10

100

1000

10000

1000000 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39

m/s

frequ

ency

correct NO missed false alarm hit

model at “face value”

Violent storm warning: gusts>= 29 m/s

183rd verification workshop, ECMWF 2007, Göber

1

10

100

1000

10000

1000000 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39

m/s

frequ

ency

correct NO missed false alarm hit

Violent storm warning: gusts>= 29 m/s

193rd verification workshop, ECMWF 2007, Göber

Bias freehit rate>90%

Violent storm warning: gusts>= 29 m/s

0

50

100

150

200

250

300

11 13 15 17 19 21 23 25 27 29 31 33 35 37

threshold in m/s

%

hit rate

false alarm ratio

Bias

Heidke skill score

203rd verification workshop, ECMWF 2007, Göber

Re-labeling„model bias = forecaster bias“

model in m/s ----> „model gust interpretationfor warnings “

13 ----> 14 (near gale)16 ----> 18 (gale)22 ----> 25 (storm)25 ----> 29 (violent storm)30 ----> 33 (hurricane force)

Verification of heavily biased model ? Quite similar to forecaster !

213rd verification workshop, ECMWF 2007, Göber

0

0,1

0,2

0,3

0,4

0,5

0,6

0,7

0,8

0,9

1

0,0 0,2 0,4 0,6 0,8 1,0

F

H

model: near gale (>14m/s)forecaster: near gale (>14m/s)no skill

Model at face value

smaller mode

l values=

overforecast

ingla

rger

mod

el v

alue

s=un

derf

orec

astin

g

forecaster

Relative Operating Characteristics (ROC)

223rd verification workshop, ECMWF 2007, Göber

0

0,1

0,2

0,3

0,4

0,5

0,6

0,7

0,8

0,9

1

0,0 0,2 0,4 0,6 0,8 1,0

F

H

model: violent stormforecaster: violent stormno skillmodel: near gale (>14m/s)forecaster: near gale

Face value

overforecasting

unde

rfor

ecas

ting

forecaster

233rd verification workshop, ECMWF 2007, Göber

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0.0 0.2 0.4 0.6 0.8 1.0

F

H

near galegalestormviolent stormhurrican forceforecaster: galeforecaster: stormforecaster: violent stormforecaster: hurrican forceno skill

243rd verification workshop, ECMWF 2007, Göber

My conclusions

End user forecast verification: face value (incl. space-time point)

Guidance verification: measure potential of the guidance usingFuzzy, Object, ........., Signal detection Theory (ROC)

Comparing peaches and apples�� On the accuracy of gust warnings issued by forecasters and the accuracy of the model guidance