Short Range Ensemble Prediction System Verification over Greece Petroula Louka, Flora Gofa

Short Range Ensemble Short Range Ensemble Prediction SystemPrediction System

Verification over GreeceVerification over Greece

Petroula Louka, Flora GofaPetroula Louka, Flora Gofa Hellenic National Meteorological ServiceHellenic National Meteorological Service

http://www.hnms.gr/hnms/greek/index_html

10th COSMO General Meeting10th COSMO General Meeting

HNMS involvement in HNMS involvement in COSMO-SREPSCOSMO-SREPS

Verification of LM-COSMO ensemble Verification of LM-COSMO ensemble forecasts for MAPD-Phase (06-12/2007)forecasts for MAPD-Phase (06-12/2007)

Cases of 72-hour forecast horizon, 16 Cases of 72-hour forecast horizon, 16 membersmembersVerification domain: GreeceVerification domain: GreeceData used: SYNOP data covering GreeceData used: SYNOP data covering GreeceParameters verified:Parameters verified:o 2m temperature2m temperatureo Mean Sea Level Pressure (MSLP)Mean Sea Level Pressure (MSLP)o PrecipitationPrecipitation

Statistical analysis of the resultsStatistical analysis of the results



COSMO-SREPS COSMO-SREPS domaindomain



Greek SYNOP stationsGreek SYNOP stations



COSMO-SREPS COSMO-SREPS members members



Statistical analysis Statistical analysis methodsmethods

For continuous parameters such as For continuous parameters such as Temperature and MSLPTemperature and MSLP

BiasBiasRMSERMSE

For non-continuous parameters For non-continuous parameters (precipitation)(precipitation)

Deterministic approach Deterministic approach o Multi-category contingency tables Multi-category contingency tables o POD, FAR, ETSPOD, FAR, ETS

Probabilistic approach (e.g., BSS, ROC Probabilistic approach (e.g., BSS, ROC diagrams, etc)diagrams, etc)



2m2m Temperature Temperature By MemberBy Member

Underestimation of Temperature. It seems that GME driven

members have better skill than the others.

T2m: RMSE

11.5

22.5

33.5

44.5

6 12 18 24 30 36 42 48 54 60 66 72Forecast time

12345678910111213141516avg

IFS GME NCEP UKMO

T2m: BIAS

-2.5-2

-1.5-1

-0.50

0.51

1.52

2.5

6 12 18 24 30 36 42 48 54 60 66 72Forecast time

12345678910111213141516avg



2m Temperature2m Temperature By MonthBy Month

Underestimation of summer temperatures June 2007 was exceptionally warm

with strong heat wavesThe maximum temperature in

Athens reached 46.2 ºC ! October to December RMSE

statistically acceptable. Different pattern between summer

and autumn/winter months

-6-5-4-3-2-1012

6 12 18 24 30 36 42 48 54 60 66 72Forecast time

BIA

S

Jun Jul Aug Sep Oct Nov Dec

0

1

2

3

4

5

6

6 12 18 24 30 36 42 48 54 60 66 72Forecast time

RM

SE




POD plots POD plots dependence on Driving Modeldependence on Driving Model

YESObsHitsPOD

POD: >0.1mm

00.10.20.30.40.50.60.70.80.9

1

6 12 18 24 30 36 42 48 54 60 66 72Forecast time

12345678910111213141516

POD: >1mm

00.10.20.30.40.50.60.70.80.9

1

6 12 18 24 30 36 42 48 54 60 66 72Forecast time

12345678910111213141516

POD: >5mm

00.10.20.30.40.50.60.70.80.9

1

6 12 18 24 30 36 42 48 54 60 66 72Forecast time

12345678910111213141516

POD: >10mm

00.10.20.30.40.50.60.70.80.9

1

6 12 18 24 30 36 42 48 54 60 66 72Forecast time

12345678910111213141516

IFS GME NCEP UKMO



POD plots POD plots dependence on convective scheme etc.dependence on convective scheme etc.

YESObsHitsPOD

POD: >0.1mm

00.10.20.30.40.50.60.70.80.9

1

6 12 18 24 30 36 42 48 54 60 66 72Forecast time

12345678910111213141516

POD: >1mm

00.10.20.30.40.50.60.70.80.9

1

6 12 18 24 30 36 42 48 54 60 66 72Forecast time

12345678910111213141516

POD: >5mm

00.10.20.30.40.50.60.70.80.9

1

6 12 18 24 30 36 42 48 54 60 66 72Forecast time

12345678910111213141516

POD: >10mm

00.10.20.30.40.50.60.70.80.9

1

6 12 18 24 30 36 42 48 54 60 66 72Forecast time

12345678910111213141516

Tiedtke1 Kain-Fritsch1 Tiedtke2 Tiedtke3



FAR plots FAR plots dependence on Driving Modeldependence on Driving Model

YESFcalarmsFalseFAR

FAR: >0.1mm

00.10.20.30.40.50.60.70.80.9

1

6 12 18 24 30 36 42 48 54 60 66 72Forecast time

12345678910111213141516

FAR: > 1mm

00.10.20.30.40.50.60.70.80.9

1

6 12 18 24 30 36 42 48 54 60 66 72Forecast time

12345678910111213141516

FAR: > 5mm

00.10.20.30.40.50.60.70.80.9

1

6 12 18 24 30 36 42 48 54 60 66 72Forecast time

12345678910111213141516

FAR: > 10mm

00.10.20.30.40.50.60.70.80.9

1

6 12 18 24 30 36 42 48 54 60 66 72Forecast time

12345678910111213141516

IFS GME NCEP UKMO



FAR plots FAR plots dependence on convective scheme etc.dependence on convective scheme etc.

YESFcalarmsFalseFAR

FAR: >0.1mm

00.10.20.30.40.50.60.70.80.9

1

6 12 18 24 30 36 42 48 54 60 66 72Forecast time

12345678910111213141516

FAR: > 1mm

00.10.20.30.40.50.60.70.80.9

1

6 12 18 24 30 36 42 48 54 60 66 72Forecast time

12345678910111213141516

FAR: > 5mm

00.10.20.30.40.50.60.70.80.9

1

6 12 18 24 30 36 42 48 54 60 66 72Forecast time

12345678910111213141516

FAR: > 10mm

00.10.20.30.40.50.60.70.80.9

1

6 12 18 24 30 36 42 48 54 60 66 72Forecast time

12345678910111213141516




Brier Skill ScoreBrier Skill Score

BSS measures the improvement of the probabilistic forecast relative to the sample climatology

The forecast system has predictive skill if BSS is positive (better than climatology), a perfect system having BSS = 1

cliBSBSBSS 1 ooBScli 1

= total frequency of the event (sample climatology)

yuncertaintyreliabilitresolutionBSS

o



BSS plotsBSS plotsJune-December 2007

24hr pred

-0.3-0.2-0.1

00.10.20.30.40.5

0 10 20 30 40

Threshold (mm/24hr)

BSS

June-December 200748hr pred

-0.3-0.2-0.1

00.1

0.20.30.40.5

0 10 20 30 40

Threshold (mm/24hr)

BSS


-0.3-0.2-0.1

00.10.20.3

0.40.5

0 10 20 30 40

Threshold (mm/24hr)

BSS

The predictive skill is good (positive BSS) for the smaller precipitation thresholds

The first day shows better scores than the other two (especially when compared to the third)

The size of the sample affects the score for the larger precipitation thresholds



BSS plots BSS plots dependence on Driving Modeldependence on Driving Model


-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0 10 20 30 40

Threshold (mm/24hr)

BSS

All mbs

IFS

GME

NCEP

UKMO


-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0 10 20 30 40

Threshold (mm/24hr)

BSS

All mbs

IFS

GME

NCEP

UKMO


-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0 10 20 30 40

Threshold (mm/24hr)

BSS

All mbs

IFS

GME

NCEP

UKMO

It seems that for the 1st day all members group together

Some members show negative BSS for low thresholds

For the 2nd and 3rd days IFS seems to provide better score

The size of the sample affects the score



BSS plots BSS plots dependence on Physical parameterisationsdependence on Physical parameterisations

Members with Tiedtke convective scheme group together

The perturbations of the particular parameters for turbulent and length scale are less important than convective schemes

It seems that the members with Kain-Fritsch convective scheme have worse performance than the others


-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5

0 10 20 30 40

Threshold (mm/24hr)

BSS

All mbs

Tdtk1

KF1

Tdtk2

Tdtk3


-0.5

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5

0 10 20 30 40

Threshold (mm/24hr)

BSS

All mbs

Tdtk1

KF1

Tdtk2

Tdtk3


-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0 10 20 30 40

Threshold (mm/24hr)

BSS

All mbs

Tdtk1

KF1

Tdtk2

Tdtk3



Reliability Reliability DiagramsDiagrams

The Frequency of an observed event is plotted against the forecast probability of the event.

If the curve lies below the 45° line, the probabilities are overestimated

Points between the "no skill" line and the diagonal contribute positively to the BSS

(resolution > reliability).

yuncertaintyreliabilitresolutionBSS




48hr48hrReliability Diagramthreshold: 0.2mm/24hr

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

Forecast Probability

Obs

erve

d Fr

eque

ncy

Climatology – No resolution

No skill

Reliability Diagramthreshold: 2mm/24hr

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1


Obs

erve

d Fr

eque

ncy


0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1


Obs

erve

d Fr

eque

ncy Overestimation of the probability

especially for the larger threshold, although

Small sample for large thresholds



Relative Operating Relative Operating CharacteristicCharacteristic

ROC is a measure of forecast skill. ROC is a tool that permits to evaluate the ability of the

forecast system to discriminate between occurrence and non-occurrence of a precipitation event (to detect the event)

It measures resolution (YES or NO event), but not reliability (e.g. biased forecast)

ROC area < 0.5 indicates no skill.



ROC AreaROC AreaJune-December 2007

24hr pred

0.60.650.7

0.750.8

0.850.9

0.951

0 10 20 30 40

Threshold (mm/24hr)

ROC


0.60.650.7

0.750.8

0.850.9

0.951

0 10 20 30 40

Threshold (mm/24hr)

ROC


0.60.650.7

0.750.8

0.850.9

0.951

0 10 20 30 40

Threshold (mm/24hr)

ROC

ROC area values are generally high for the lower thresholds

The ensemble can discriminate those events

It seems that for the first predictive period (24hr) ROC area has larger values for more precipitation thresholds (up to 15mm/day) compared with the other two periods



ROC AreaROC Areadependence on Driving Modeldependence on Driving Model


0.5

0.6

0.7

0.8

0.9

1

0 10 20 30 40

Threshold (mm/24hr)

RO

C

All mbs

IFS

GME

NCEP

UKMO


0.5

0.6

0.7

0.8

0.9

1

0 10 20 30 40

Threshold (mm/24hr)

ROC

All mbs

IFS

GME

NCEP

UKMO


0.5

0.6

0.7

0.8

0.9

1

0 10 20 30 40

Threshold (mm/24hr)

RO

CAll mbs

IFS

GME

NCEP

UKMO

In general all members contribute similarly

Small sample for larger thresholds



ROC AreaROC Areadependence on Physical parameterisationsdependence on Physical parameterisations


0.5

0.6

0.7

0.8

0.9

1

0 10 20 30 40

Threshold (mm/24hr)

ROC

All mbs

Tdtk1

KF1Tdtk2

Tdtk3


0.5

0.6

0.7

0.8

0.9

1

0 10 20 30 40Threshold (mm/24hr)

RO

C

All mbs

Tdtk1

KF1Tdtk2

Tdtk3


0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 10 20 30 40Threshold (mm/24hr)

RO

CAll mbs

Tdtk1

KF1Tdtk2

Tdtk3

Members with Tiedtke convective scheme group together

The perturbations of the particular parameters for turbulent and length scale are less important than convective schemes

The members with Kain-Fritsch convective scheme seem to perform better

=> better resolution (but not good reliability, c.f. BSS diagrams)



ROC CurvesROC Curves

Hit rates are plotted against the corresponding false alarm rates to generate the ROC Curve.

The area under the ROC curve is used as a statistical measure of forecast usefulness.

event theof soccurrence-non ofnumber totalevent theof forecastscorrect non ofnumber

dbbF

event theof soccurrence ofnumber totalevent theof forecastscorrect ofnumber

caaH

a+b+c+d=nb+da+cTotal

c+ddcNo

a+bbaYes

TotalNoYes

Event observedEventForecast

a+b+c+d=nb+da+cTotal

c+ddcNo

a+bbaYes

TotalNoYes

Event observedEventForecast



ROC CurvesROC CurvesROC Curve

threshold: 0.2mm/24hr

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

False Alarm Rate

Hit

Rat

e

ROC Curvethreshold: 0.5mm/24hr

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

False Alarm Rate

Hit

Rat

e

ROC Curvethreshold: 1mm/24hr

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

False Alarm Rate

Hit

Rat

e


0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

False Alarm Rate

Hit

Rat

e


0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

False Alarm Rate

Hit

Rat

e


0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

False Alarm RateH

it R

ate



Remarks Remarks on Temperatureon Temperature

Temperature is strongly (~5°C) underestimated during the summer months

Both Bias and RMSE exhibited a diurnal cycle with the (absolute) maxima being during the hottest summer hours, while for the autumn/winter months the diurnal variation had the opposite behaviour

The initial and boundary condition perturbations contribute to the BIAS and RMSE more than the physical parameter perturbations



Remarks Remarks on Precipitationon Precipitation

Precipitation amount is overestimated It is not evident a consistent attitude for the

forecasted precipitation driven by a certain initial and boundary condition model

Perturbations of the convective schemes are more important than the perturbations of the particular parameters for turbulent and length scales used

It seems that the members with Kain-Fritsch convective scheme have better resolution but worse reliability compared to the members with Tiedtke convective scheme





MSLPMSLP By MonthBy Month

-3

-2

-1

0

1

2

3

6 12 18 24 30 36 42 48 54 60 66 72Forecast time

BIA

S


0

0.51

1.5

2

2.53

3.5

4

6 12 18 24 30 36 42 48 54 60 66 72Forecast time

RM

SE




2m2m Temperature Temperature By MemberBy Member

Underestimation of Temperature.

No particular effect of the scaling parameters is evident.

T2m: BIAS

-2.5-2

-1.5-1

-0.50

0.51

1.52

2.5

6 12 18 24 30 36 42 48 54 60 66 72Forecast time

12345678910111213141516avg

T2m: RMSE

1

1.5

2

2.5

3

3.5

4

4.5

6 12 18 24 30 36 42 48 54 60 66 72Forecast time

12345678910111213141516avg






0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1


Obs

erve

d Fr

eque

ncy


No skill


0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1


Obs

erve

d Fr

eque

ncy


0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1


Obs

erve

d Fr

eque

ncy





0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1


Obs

erve

d Fr

eque

ncy


No skill


0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1


Obs

erve

d Fr

eque

ncy


0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1


Obs

erve

d Fr

eque

ncy


Date post:	19-Mar-2016
Category:	Documents
Upload:	holt
View:	33 times
Download:	0 times

Short Range Ensemble Prediction System Verification over Greece Petroula Louka, Flora Gofa

Documents