New TAF verification

Version 1.41 Nordic

Working group:

Alberto Blanco S., Matti Astevirta, Timo Erkkilä, Matti Heinonen, Mikael Jokimäki, Tero Lassila, Ilpo

Snellman, Raimo Pukkila, Kari Österberg

Table of contents

• TAF verification slide 3

• Table for values: Visibility and Ceiling 4

• Verification limits for wind 5

• Verification of weather 6

• TEMPO, a problem 8

• Verification of PROB-groups 10

• Examples 11

• Verification of Persistencies and BECMG-counter 24

• Tafs Overdue%,Vis and Clouds amendlength 25

• Verification html-filenames 26

3

TAF verification

• Verification even for single TAF

• Every TAF graded in 30 min periods

• TAF gets fingerprints

• TAF compared to observation (METAR, Auto-METAR)

• Good points for TAF if in the adjacent class of the observation

• All lengths of TAF:s, which starts at 00, 06, 09, 12, 15, 18 or 21 UTC are verified

• Parameters

– Cloud base (Clouds)

– Visibility (Vis)

– Wind direction (WD) 10/0

points (right/wrong)

– Wind velocity (WS) 10/0

points (right/wrong)

– Weather FAR (False Alarm

Ratio), POD (Probability Of

Detection), Hit rates

4

Table for values:

Visibility and Ceiling• The further forecast is from the observation

the less points it gets

• Point will be calculated for ½ hour observations

• Max points 10/obs,

• Annex 3 classes will be used

• CAVOK, NSC, SKC, FEW and SCT are verified as 1500 ft BKN- and OVC-clouds

• In Ceiling verification Vertical Visibility (VV) is verified as Ceiling

• Used Visibility limits are 150 m, 350 m, 600 m, 800 m, 1500 m, 3000 m and 5000 m

• Used Ceiling limits are 100 ft, 200 ft, 500 ft, 1000 ft and 1500 ft

• Forecasts immediately before FM timegroup can be either in the previous or following basic Visibility or Ceiling.

• BECMG- and TEMPO: look examples

Ceil

ing

Visib

ility

Class

diff.

0 10 10

1 7 7

2 1 1

3 0 0

4 0 0

5

Verification limits for wind

• Wind direction 60 , observation min 10 knots for verification

• Wind speed 10 knots, observation min 3 knots for verification.

• If between above limits, Wd and Ws 10 points, otherwise 0 points (correct/wrong)

• Basic forecast, FM and BECMG are verified. (TEMPO and PROB are not verified). FM is verified after the time of change as new basic weather. During BECMG time period both preceeding and following forecasts are compared, if either is observed → 10 points (correct).

• Every TAF graded in 30 min perioids

• During BECMG time period both preceeding and following forecasts are compared, if either is observed → 10 points (correct).

6

Verification of weather

• Weather is verified in following groups:

• SNOW: - SN, +SN, SNRA, +SNRA, RASN, +RASN, SHRASN, +SHRASN, BLSN, +BLSN, SG and +SG.

- All weathers, which contains SHSN or +SHSN, but not –SHSN.

- IC and +IC, if observed visibility is <= 5000 m.

• RAIN: RA, +RA, SHRA, +SHRA, SNRA, +SNRA, RASN , +RASN, SHSNRA, +SHSNRA, DZ and +DZ.

- All weathers, which contains SHRA or +SHRA, but not –SHRA.

• ICE: -FZDZ, FZDZ, +FZDZ, -FZRA, FZRA, +FZRA, FZFG and +FZFG.

• THUNDER: TS, +TS, -TSRA, TSRA, +TSRA, TSGR and +TSGR.

• GRANULE: GR, +GR, GS, +GS, PL and +PL.

• Significant weathers which are verified in own groups (e.g. DS and +DS are verified in same group):

-(+)DRDU, (+)DRSA, (+)BLDU, (+)BLSA, (+)DS, (+)SS, (+)SQ, (+)FC, (+)VA and (+)PO.

-(+)FU, (+)SA, (+)DU and (+)HZ, if observed visibility is <= 5000 m.

• All other weathers are verified as no significant weather (NSW).

• Verification in 1 hour periods. If forecast and observation is in same group forecast gets 10 points (correct).

• If no significant weather is forecasted and either observation within the hour shows the significant weather, hour gets 0 point (wrong).

7

Verification of weather

• One hour period begins on the hour. Observations made 20 min past and 50 min past belong to this hour period. To first TAF hour it belongs also10 min before observation.

• For alternative forecast TEMPO time period, both basic and alternative forecasts are used e.g. observed weather is found in either forecast, the forecast gets 10 points (correct. )

• FM is verified after the time of change as new basic weather. Forecasts immediately before FM timegroup can be either in the previous or following basic weather.

• During BECMG time period both preceeding and following forecasts are compared, if either is observed → 10 points (correct).

• If it is not forecasted and not observed weather -> 10 points (correct).

• POD counts how many of observations is forecasted in 1 hour perioids.

• FAR counts number of wrong forecasts. If not one of forecasts (Basic, Becmg, Fm or Tempo) is not found in any metar within the hour -> forecast in this TAF-hour is wrong -> If TAF length is 9 hour ->FAR gets 1/9 points.

• HitRate is weather points for TAF divided with 10. E.g. weather points 8.9 -> HitRate for TAF is 0.9.

8

TEMPO, a problem

• According to ANNEX 3, TEMPO group describes fluctuations in the weather. During TEMPO both basic weather group (eg. Normal TAF forecast) or the alternative weather (eg. TEMPO group) can be correct.

• ANNEX 3: ”a period of one hour in each instance and, in the aggregate cover less than one half of the forecast period during which the fluctuations are expected to occur ”

• In verification TEMPO valid time will be divided in 3 hour periods.

• During TEMPO any forecast can agree with the observation. If TEMPO will not occur, atleast one forecast is incorrect during 3 hour period. Forecast closest to the observation will be interpreted as incorrect. Eg. minimum error. (And maximum points)

• FM is verified after the time of change as new basic weather.

9

TEMPO, a problem

• When basic or alternative weather does not occur (above basic, above alternative, below basic, below alternative)→ incorrectforecast

• Comparison is eiher to basic or alternative weather whichever is closest. No more than 2 subsequent alternative weather AND lessthan half of the alternative forecast period can be compared (to observation). Otherwise basic weather is compared (to observation).

• Atleast every 3rd basic weather forecast will be compared to the observation. (In case alternative does not occur and observationsare in the basic group)

• Comparison in 3 hour periods for alternative weather (TEMPO). IfTEMPO does not occur comparison is made to observation closest to TEMPO and not in basic weather. Atleast every 3rd is compared to basic weather and less than half to alternative (TEMPO) weather

10

Verification of PROB-groups

• Prob-groups are verified to own tables, Prob30 and Prob40 groups has own tables.

• Prob30 and Prob30 Tempo are verified as Prob30 group.

• Prob40 and Prob40 Tempo are verified as Prob40 group.

• Ceiling and visibility verification:

- If forecast is in same gategory as observation, prob-group gets 100 % hit rate to this metartime, otherwise hit rate is 0 % to this metartime.

• Wind direction and wind speed verification:

-Wind direction 60 , observation min 10 knots for verification

-Wind speed 10 knots, observation min 3 knots for verification.

-If between above limits, Wd and Ws gets hit rate100 % points (correct) to this metartime, otherwise hit rate is 0 % (wrong) to this metartime.

• Weather verification:

-Weather in prob-groups are verified as basic weather in one hour perioids. -If forecasted and observed weathers are in same group, this tafhour gets hit rate 100 % points (correct) to this tafhour, otherwise hit rate is 0 % (wrong) to this tafhour.

11

Examples

• In these examples only one parameter

(visibility ) is used for clarity.

• In verification points are given to 4

parameters: ceiling, visibility, wind

direction and wind speed.

• An average of these is calculated and this

gives the final points for each TAF

12

Comparison to basic forecast AND only 2 subsequent to

alternative group (TEMPO) if alternative occurs

261524 09007KT 5000 -RASN SCT001 OVC003 TEMPO 1524

1500 BR OVC001=

km 0 (15) 1 (16) 2 3 4 5 6 7 8 9 (24)

8

5

3

1,5

0,8

0,60,35

Points 10 7 7 10 10 0 10 10 1 7 7 7 7 7 7 7 7 7 1 6,8 Sum

observation

basic forecast Compared

to basic forecast

Compared

To alternative forecastalternative

forecast

Alternative (TEMPO) occurs and is long

13

261524 09007KT 5000 -RASN SCT001 OVC003 TEMPO 1524

1000 BR OVC001= note different forecast from previous slide, alternative does not occur.

Real forecast and observations at EFRO

km 0 (15) 1 (16) 2 3 4 5 6 7 8 9 (24)

8

5

3

1,5

0,8

0,60,35

Points 10 7 7 7 7 7 1 7 7 7 7 7 7 7 7 7 7 7 0 6,3 Sum

observation

basic forecast

alternative

forecast

Compared

to basic forecast

Compared

To alternative forecast

Alternative (TEMPO) does not occur and is long

Comparison to basic forecast AND alternative group (TEMPO)

if alternative does not occur

14

Alternative compared in 3 hour periods. If alternative occurs every third is

compared to basic. If does not occur compared to whichever is closest and not in

basic forecast AND atleast one is compared to alternative in 3 hour perioids AND

no more than 2 subsequent AND less than half is compared to alternative.

261524 09007KT 5000 -RASN SCT001 OVC003 TEMPO 1524 1000 BR OVC001=

km 0 (15) 1 (16) 2 3 4 5 6 7 8 9 (24)

8

5

3

1,5

0,8

0,60,35

Points 10 7 7 7 7 7 1 7 7 7 7 7 7 7 7 7 7 7 0 6,5 Sum

observation

basic forecast

alternative

forecast

Compared

to basic forecast

Compared

To alternative forecast

Alternative (TEMPO) does not occur and is long

Min 1 alt./3 hours AND closer to alt.

Closer to alt.

Closer to basic Closer to basic

Closer to basic

Min 1 alt./3 hours

15

Alternative compared in 3 hour periods. If alternative occurs every third is

compared to basic. If does not occur compared to whichever is closest and not in

basic forecast AND atleast one is compared to alternative in 3 hour perioids AND

no more than 2 subsequent AND less than half is compared to alternative.

261524 09007KT 5000 -RASN SCT001 OVC003 TEMPO 1521 1000 BR OVC001=

km 0 (15) 1 (16) 2 3 4 5 6 7 8 9 (24)

8

5

3

1,5

0,8

0,60,35

Points 10 7 7 7 7 7 1 7 7 7 7 7 7 7 7 7 7 7 10 7,0 Sum

observation

basic forecast

alternative

forecast

Compared

to basic forecast

Compared

To alternative forecast

Shorter (6h) alternative

Min 1 alt./3 hours AND closer to alt.

Closer to alt.

Closer to basic Closer to basic

Closer to basic

16

Alternative compared in 3 hour periods. If alternative occurs every third is

compared to basic. If does not occur compared to whichever is closest and not in

basic forecast AND atleast one is compared to alternative in 3 hour perioids AND

no more than 2 subsequent AND less than half is compared to alternative.

261524 09007KT 5000 -RASN SCT001 OVC003 TEMPO 1518 1000 BR OVC001=

km 0 (15) 1 (16) 2 3 4 5 6 7 8 9 (24)

8

5

3

1,5

0,8

0,60,35

Points 10 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 10 7,3 Sum

observation

basic forecast

alternative

forecast

Compared

to basic forecast

Compared

To alternative forecast

Still shorter alternative (3h)

Closer to alt.

Closer to basic Closer to basicCloser to basic

17

No alternative forecast

261524 09007KT 5000 -RASN SCT001 OVC003 =

km 0 (15) 1 (16) 2 3 4 5 6 7 8 9 (24)

8

5

3

1,5

0,8

0,60,35

Points 10 7 7 1 1 7 7 7 7 7 7 7 7 7 7 7 7 7 10 6,7 Sum

No TEMPOobservation

basic forecast

18

TEMPO, BECMG, TEMPO real forecast and obs. at EFRO

280615 22005KT 0300 FG VV002 TEMPO 0609 4000 BR BKN004

BECMG 0911 8000 –SHSNRA BKN013 TEMPO 1115 4000 BKN008 =

km 0 (06) 1 (07) 2 3 4 5 6 7 8 9 (15)

8

5

3

1,5

0,8

0,60,35

Points 10 10 10 10 1 1 10 10 10 10 10 1 10 10 10 10 10 10 10 8,6 Sum

BECMG between basic and alternative →10 pointsobservation

basic forecast Compared

to basic forecast

Compared

To alternative forecast

Closer to basic

Closer to alt.

Between basic

and alt.

Min 1 alt./3 hours

19

Below alternative, above basic and ”between”

261524 09007KT 5000 -RASN SCT001 OVC003 TEMPO 1524

1500 BR OVC001=

km 0 (15) 1 (16) 2 3 4 5 6 7 8 9 (24)

8

5

3

1,5

0,8

0,60,35

Points 10 7 7 10 10 0 10 10 1 7 7 7 7 7 7 7 7 7 1 6,8 Sum

observation

basic forecast Compared

to basic forecast

Compared

To alternative forecastalternative

forecast

Alternative (TEMPO) occurs and is long

20

Above alternative

km 0 (15) 1 (16) 2 3

8

5

3

1,5

0,8

0,60,35

Above alternative

km 0 (15) 1 (16) 2 3

8

5

3

1,5

0,8

0,60,35

Above alternative

Alternative worse than basic Alternative better than basic

observation

basic forecast

alternative

forecast

21

Below alternative

261524 09007KT 5000 -RASN SCT001 OVC003 TEMPO 1524

1500 BR OVC001=

km 0 (15) 1 (16) 2 3 4 5 6 7 8 9 (24)

8

5

3

1,5

0,8

0,60,35

Points 10 7 7 10 10 0 10 10 1 7 7 7 7 7 7 7 7 7 1 6,8 Sum

observation

basic forecast Compared

to basic forecast

Compared

To alternative forecastalternative

forecast

Alternative (TEMPO) occurs and is long

Below alternative

22

Above basic

261524 09007KT 5000 -RASN SCT001 OVC003 TEMPO 1524

1500 BR OVC001=

km 0 (15) 1 (16) 2 3 4 5 6 7 8 9 (24)

8

5

3

1,5

0,8

0,60,35

Points 10 7 7 10 10 0 10 10 1 7 7 7 7 7 7 7 7 7 0 6,7 Sum

observation

basic forecast Compared

to basic forecast

Compared

To alternative forecastalternative

forecast

Alternative (TEMPO) occurs and is long

Above basic

23

Below basic

261524 09007KT 5000 -RASN SCT001 OVC003 =

km 0 (15) 1 (16) 2 3 4 5 6 7 8 9 (24)

8

5

3

1,5

0,8

0,60,35

Points 10 7 7 1 1 0 1 1 1 7 7 7 7 7 7 7 7 7 10 5,4 Sum

observation

basic forecast

Below basic

24

Verification of Persistencies and

BECMG-counter

• Persistency-taf is generated from metar, which is made 40 minutes before tafs start (from observationtime (h-1):20 metar).

• If this metar does not exist, then persistency-taf is generated from metar, which is made 1 hour and 10 minutes before tafs start (from observationtime (h-2):50 metar).

• html-filename includes word fcpersistency or word ftpersistency

• BECMG-counter, which is in column BECMG in verificationtables counts how many BECMG-groups are in one taf, in one meteorologists all tafs or in all tafs in the verificationtable, depending on where BECMG-counter locates. This feature was added starting from November 2007.

25

Tafs Overdue%,Vis and Clouds

amendlength• All following features are only in html-files, in which all tafs are verified, for definition look next

page.

• Tafs overdue%-counter is RRA-tafs (=overdued Tafs) piece of RRA-tafs+Normaltafs. Overdue%=(RRA-tafs)/(RRA-tafs+Normaltafs).

• VisFix and ClFix are amendlengths in minutes for parametres Visibility and CloudsBase.Maximal amendlength starts from tafs starttime and ends 40 minutes before next tafs starttime.

• So maximal amendlength for fcTafs is 140 minutes and for ftTafs 320 minutes.

• Assumption is that wrong forecast must be amended if metar-observation is in smaller category than in any tafs group.

• In Taf forecast can be in any group: Basic, Becmg, Tempo or Prob.

• New need of amend starts, when difference between taf- and metar-classes are bigger than between previous metars and tafs classes and metar-observation is in smaller category than in any tafs group.

• Amendlengths VisFix and CloudsFix are sum of amendlengths of each taf.

• NoViFix ja NoClFix are count of forecasts, which are not amended, although those ought to amended.

• In counting of VisFix, ClFix, NoViFix and NoClFix are always used latest achievable amendtaf.

• These features were added 5. day of February 2008.

• From beginning of February 2008 is added feature, which counts number of verified amendtafs.

26

Verification html-filenames

• html-filename e.g.: 200707_fctaf_essa_All_Prob.html means:

• 200707 means that verified year is 2007 and verified month is 07

• fctaf is verified taftype

• essa is verified stations Icao. If filename does not include Icao, then Allstations tafs, which are verified for that country is verified to same html-file.

• All means that all tafs are verified: amendtafs, delayed tafs and syntaxcorrected tafs. If html-filename does not include All-word, then only first taffor every taftime is verified. If trend includes BECMG-group, only last metarin two hour trend-time is verified

• Prob means that only tafs prob-groups are only verified. If html-filenamedoes not include Prob-word then Prob-group are not verified, but Basic and other groups are verified.

• If html-filename includes 2000ft, then used Ceiling limits are 100 ft, 200 ft, 500 ft, 1000 ft, 1500 ft, 2000 ft and used Visibility limits are 150 m, 350 m, 600 m, 800 m, 1500 m, 3000 m, 5000 m and 8000 m. These limits are onlyused in some Denmarks ICAO-stations. This feature was added frombeginning of year 2008.