NWS TAF VerificationBrandi Richardson

NWS Shreveport, LA

Home

Previous

Next

Help

Do we care how our forecasts verify?

NO!

Home

Previous

Next

Help

Do we care how our forecasts verify?

• Yes!• The NWS measures verification by many

means– Probability of Detection (POD)– False Alarm Ratio (FAR)– Critical Success Index (CSI)– Percent Improvement

• Set goals for verification• Local offices add own flavor• Total IFR (IFR, LIFR, VLIFR)

Home

Previous

Next

Help

Why is verification important?

• Need to know what to improve– Lose credibility if too many forecasts are

wrong• Lose customers• Lose jobs

– Additional training– New techniques– Improved model guidance

• Need to know what we are doing well

Home

Previous

Next

Help

NWS TAF Verification

• TAFs evaluated 12 times per hour (every five minutes), or 288 times per 24-hour period

• TAFs compared to ASOS five-minute observations – ASOS = Automated Surface Observing

System, located at TAF airports• Stats calculated by flight category

– i.e., VFR, MVFR, IFR, LIFR, VLIFR

Home

Previous

Next

Help

Probability of Detection

• How often did we correctly forecast a particular flight category to occur?– Also known as “Accuracy”

• POD = V/(V+M)– V = forecasted and verified events

• Ex: IFR conditions forecasted…IFR conditions occurred– M = missed events

• Ex: VFR conditions forecasted…IFR conditions occurred• Ranges from 0 – 1, 1 being perfect

Home

Previous

Next

Help

False Alarm Ratio

• How often did we forecast a particular flight category to occur that did not occur?– i.e., how often did we “cry wolf”?

• FAR = U/(U+V)– U = forecasted and unverified

• Ex: IFR forecasted…VRF occurred– V = forecasted and verified events

• Ex: IFR conditions forecasted…IFR conditions occurred• Ranges from 0 – 1, 0 being perfect

Home

Previous

Next

Help

Critical Success Index

• CSI = V/(V+M+U)– V = forecasted and verified events

• Ex: IFR conditions forecasted…IFR conditions occurred– M = missed events

• Ex: VFR conditions forecasted…IFR conditions occurred– U = forecasted and unverified

• Ex: IFR forecasted…VRF occurred• Ranges from 0 – 1, 1 being perfect• Incorporates both POD and FAR• Overall score of performance

Home

Previous

Next

Help

Percent Improvement

• Forecaster CSI vs. Model Guidance CSI– Did we beat the model?

IFR will prevail…

IFR?! It’s July and dew points are in

the 20s! Take that!

Forecaster

GFS

Home

Previous

Next

Help

2009 NWS Goals

• The NWS has set goals for TAF forecasts– For total IFR (includes IFR, LIFR, and

VLIFR)• POD ≥ 0.640 (64%)• FAR ≤ 0.430 (43%)

• How do we measure up?...

Home

Previous

Next

Help

Examples of Local TAF Verification

4th qtr 05

1st qtr 06

2nd qtr 06

3rd qtr 06

4th qtr 06

1st qtr 07

2nd qtr 07

3rd qtr 07

4th qtr 07

1st qtr 08

2nd qtr 08

3rd qtr 08

4th qtr 08

1st qtr 09

2nd qtr 09

3rd qtr 09

4th qtr 09

2005 2006 2007 2008 2009

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1.000

IFR Probability of Detection (POD) by QuarterSRSHVSHV GFS MOSGPRA Goals

Perc

ent

Home

Previous

Next

Help

Examples of Local TAF Verification

4th

qtr 0

51s

t qtr

062n

d qt

r 06

3rd

qtr 0

64t

h qt

r 06

1st q

tr 07

2nd

qtr 0

73r

d qt

r 07

4th

qtr 0

71s

t qtr

082n

d qt

r 08

3rd

qtr 0

84t

h qt

r 08

1st q

tr 09

2nd

qtr 0

93r

d qt

r 09

4th

qtr 0

91s

t qtr

102n

d qt

r 10

3rd

qtr 1

04t

h qt

r 10

1st q

tr 11

2nd

qtr 1

13r

d qt

r 11

4th

qtr 1

11s

t qtr

122n

d qt

r 12

3rd

qtr 1

24t

h qt

r 12

1st q

tr 13

2nd

qtr 1

33r

d qt

r 13

4th

qtr 1

3

2005

2006 2007 2008 2009 2010 2011 2012 2013

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1.000

IFR False Alarm Ratio (FAR) by QuarterSRSHVSHV GFS MOSGPRA Goals

Perc

ent

Home

Previous

Next

Help

Examples of Local TAF Verification

4th

qtr 0

51s

t qtr

062n

d qt

r 06

3rd

qtr 0

64t

h qt

r 06

1st q

tr 07

2nd

qtr 0

73r

d qt

r 07

4th

qtr 0

71s

t qtr

082n

d qt

r 08

3rd

qtr 0

84t

h qt

r 08

1st q

tr 09

2nd

qtr 0

93r

d qt

r 09

4th

qtr 0

91s

t qtr

102n

d qt

r 10

3rd

qtr 1

04t

h qt

r 10

1st q

tr 11

2nd

qtr 1

13r

d qt

r 11

4th

qtr 1

11s

t qtr

122n

d qt

r 12

3rd

qtr 1

24t

h qt

r 12

1st q

tr 13

2nd

qtr 1

33r

d qt

r 13

4th

qtr 1

3

2005

2006 2007 2008 2009 2010 2011 2012 2013

0.000

0.100

0.200

0.300

0.400

0.500

0.600

IFR Critical Success Index (CSI) by QuarterSR

SHV

SHV GFS MOS

Perc

ent

Home

Previous

Next

Help

The Bottom Line

• Sometimes we do get the forecast wrong.

• Examination of TAF verification statistics helps to find our weaknesses and allows us to find ways to improve our forecasts.

• The NWS strives to provide quality products and services to our aviation customers and partners.