© Crown copyright Met Office
WAFC turbulence and Cb hazard verificationRecent results and future plansDr Philip G Gill
WAFSOPSG 7/14, 30th April 2013
© Crown copyright Met Office
Contents
This presentation covers the following areas
• Introduction
• Turbulence verification
• Cb verification
• Future plans
• Questions and answers
© Crown copyright Met Office
Introduction
• Turbulence and Cb are a major cause of aviation incidents
• Turbulence and Cb hazard forecasting are part of the World Area Forecast Centre (WAFC) service provided by WAFC London and WAFC Washington
• Forecasts currently produced both by forecasters – Significant Weather Chart (SIGWX) and from gridded model output
• Objective verification was set up in 2009
• Forecasts are global but verification is currently limited
Photos © P Gill
© Crown copyright Met Office
Objective turbulence verification
© Crown copyright Met Office
Verification
• Objective verification of gridded clear air turbulence (CAT) forecasts
• Verification by severity of turbulence (eg. moderate or greater)
• Verify against automated aircraft observations from the Global Aircraft Data Set (GADS)
• Restricted to cruise level (above 28, 000 ft) to limit unwanted effects on observations of manoeuvring at lower levels
© Crown copyright Met Office
Global Aircraft Data Set
• Fleet of Boeing 747-400 aircraft
• Global coverage, but flights mainly over northern hemisphere
• Automated aircraft observations available every 4 seconds
10-19 January 2009
Good coverage of N Atlantic, US and Europe
Poor coverage of E Asia/Pacific region and southern hemisphere – need more observations
© Crown copyright Met Office
Turbulence observations
• Pilot Reports (PIREPS) are issued by pilots when turbulence is encountered – useful additional information but are subjective, aircraft specific, temporal and spatial errors, no routine null reports
• AMDAR are automated reports from aircraft but are not regular and can differ between aircraft
• Derived Equivalent Vertical Gust (DEVG) - Indicator of turbulence derived from automated in-situ flight data: vertical acceleration, aircraft mass, altitude and airspeed
• Eddy Dissipation Rate (EDR) also a measurement of turbulence derived from automated in-situ data – ICAO standard
At present access to EDR data limited to US – DEVG only option for wider coverage
© Crown copyright Met Office
Verification methodology
Aircraft track within time window
Turbulent event
Turbulence forecast field
Ellrod TI1
© Crown copyright Met Office
Forecast assessment
• Turbulent/non turbulent event defined on 10min aircraft track ~120km - approx grid size
• Forecast turbulent event –
CAT potential >= Threshold
• Observed (moderate or greater) turbulent event - DEVG>=4.5m/s
• Construct 2x2 contingency tables for each threshold
• Sum entries in contingency tables over the verification period
Turbulence observed
No turbulence observed
Turbulence forecast
Hit False alarm
No turbulence forecast
Miss Correct rejection
2x2 contingency table
Forecast skill
• Contingency tables can be used for various probability thresholds to produce a reliability table
• From the reliability table various scores can be calculated
• The skill of the forecast at discriminating between events and non-events can be measured using the Receiver Operating Characteristic (ROC) by plotting hit rate against false alarm rate
• The curve from a probabilistic forecast and single point from a deterministic forecast can be compared.
ROC Curve
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.2 0.4 0.6 0.8 1
False alarm rate
Hit
rat
e
Low threshold
High threshold
Line of no skill
Perfect forecast
Area under curve measures skill
© Crown copyright Met Office
WAFC CAT verificationRobert Coulson
Latest results – mean CAT
ROC for global WAFC max CAT forecasts against GADS data for moderate or greater turbulence Nov 2011-Oct 2012
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.2 0.4 0.6 0.8 1
False alarm rate
Hit
ra
te
WAFC blended max CAT
Acceptable hit rates?
Acceptable false alarm rates?
Area of interest could be measured by partial area under ROC curve
Latest results – max CAT
ROC for global WAFC mean CAT forecastsagainst GADS data for moderate or greater turbulence Nov 2011-Oct 2012
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.2 0.4 0.6 0.8 1
False alarm rate
Hi r
ate
WAFC blended mean CAT
Latitudinal variation
50N to 90N
20S to 20N
20N to 50N
50S to 20S
Good performance from above 20N
Summary of CAT verification
• Blending process continues to produce both consistent and skilful turbulence forecasts.
• Some variation in skill by latitude with best performance in the mid latitudes.
• Rolling 12-month results produced monthly since June 2012.
© Crown copyright Met Office
WAFC Cb verificationTeresa Hughes
Cb verification
• Verification against lightning strikes (SFERICS) from Met Office ATDnet
• Gridded SFERICS field produced using observations at +/- 30 minutes from forecast validity time
• T+24 forecast verified from all four model runs
ATDnet Domain
Latest results
ROC for global WAFC Cb forecasts against SFERICS lighning data Nov 2011-Oct 2012
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.2 0.4 0.6 0.8 1
False alarm rate
Hi r
ate
WAFC blended Cb
Latitudinal variationROC for global WAFC Cb forecasts against SFERICS lightning data
Nov 2011-Oct 2012 (50N - 80N)
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.2 0.4 0.6 0.8 1
False alarm rate
Hi r
ate
WAFC Blended Cb
ROC for global WAFC Cb forecasts against SFERICS lightning data Nov 2011-Oct 2012 (20N - 50N)
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.2 0.4 0.6 0.8 1
False alarm rate
Hi r
ate
WAFC Blended Cb
ROC for global WAFC Cb forecasts against SFERICS lightning data Nov 2011-Oct 2012 (20S - 20N)
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.2 0.4 0.6 0.8 1
False alarm rate
Hi r
ate
WAFC Blended Cb
ROC for global WAFC Cb forecasts against SFERICS lightning data Nov 2011-Oct 2012 (40S - 20S)
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.2 0.4 0.6 0.8 1
False alarm rate
Hi r
ate
WAFC Blended Cb
50N to 80N 20N to 50N
20S to 20N 40S to 20S
Summary of Cb verification
• Blending process continues to produce both consistent and skilful Cb forecasts.
• Some variation in skill by latitude
• Rolling 12-month results produced quarterly since June 2012.
© Crown copyright Met Office
Future verification plans
Improving turbulence verification
• Switch from DEVG to EDR
• ICAO standard
• Forecasting and observing EDR
• Source more EDR observations from other airlines to increase the area of coverage
• Increase coverage to fill gaps in current GADS database
Improving Cb verification
• Extend verification against lightning reports by outside of the ATDnet domain
• Source additional lightning reports outside [80N, 40S, 100W, 80E]
• Use geostationary satellite data in addition to lightning data
• MSG [60N, 60S]
• Overshooting convection product (work underway)
• New satellites over America (GOES-R) and East Asia (Himawari-8)
• Verify Cb cloud top height and Cb extent
MSG - the EuropeanGeostationary Imager
•Scans Africa, Europe, Middle East and Atlantic every 15 minutes
•3/1 km spatial resolution at 0° N, 0° E
•12 channels (different wavelengths)
•Meteosat-10 operational, Meteosat-8,-9 backup
Severe convection product
•Data from Meteosat Second Generation
•Calculated from the visible and infrared channels.
•Generated every 15 mins.
•Current product over Europe identifies area where severe convection is occurring (day-time only)
•New overshooting convection product will identify intense convective areas with overshooting cloud tops.
•Could be merged with lightning data from ATDnet
•Could be merged later with Meteosat third generation lightning data
Cloud top height product
• Data from Meteosat Second Generation
• Calculated from the infrared channels at wavelengths of 10.8, 12.0 and 13.4 microns.
• Generated every 15 mins
• Covers Europe, Atlantic, Africa
• Extend to global coverage following GOES-R and Himawari-8 satellite launches (~ 2015)
• Mask with Cb product
Verification – all forecasts
• Verify probabilistic forecasts using suitable metrics
• Forecast skill – ROC (partial area under curve?)
• Reliability diagram
• Value – relative economic value measure
• Publish verification results – plots and contingency tables
• Breakdown into WMO regions
• Forecast ranges T+12, T+24, T+36
• Rolling 12-month scores
• Geostationary satellite data limited to 60N, 60S. Polar orbiter data could supplement this if necessary.
Timescales
• 2013/14
• Preliminary work on satellite data specifications
• Routine CAT and Cb verification published by WMO region.
• 2014/15
• Include satellite data in Cb verification
• Develop Cb cloud top height verification
• Move to using EDR in turbulence verification
• 2015/16
• Expand coverage of Cb verification using new satellite data over East Asia (Himawari-8)
• 2016/17
• Expand coverage of Cb verification using new satellite data over America (GOES-R)
© Crown copyright Met Office
Questions & answers