East Coast vs. West Coast: A Documentation of Model Forecast
Failures for Eta, NAM, GFS, GEM, and ECMWF
Garrett WedamLynn McMurdie, Cliff Mass
72-hour NAM Forecast: Ridging 1024 mb contour
72-hr NAM forecast with Obs verification Valid Dec 24, 2006
24
72-hour Reality: Low Center
00-hr NAM analysis and obs verification Dec 24, 2006
1000 mb contour
• 72-hour forecast errors– Eta/NAM: 24.1 mb– CMC-GEM: 14.4 mb– ECMWF: 10.0 mb– GFS: 3.2 mb
• 48-hour forecast errors– Eta/NAM: 13.7 mb– CMC-GEM: 9.1 mb– ECMWF: 7.6 mb– GFS: 1.5 mb
• 24-hour forecast errors– Eta/NAM: 4.7 mb– CMC-GEM: 2.4 mb– ECMWF: 2.7 mb– GFS: -0.2 mb
GFS 72-hour forecast verifying 12-26
Verifying buoy
72-hour Forecast: Low Center
72-hr NAM forecast with Obs verification Valid Dec 24, 2006
992 mb contour
24
-24
-20
-27
-31
27
25
21
-8
72-hour Reality: Ridging
00-hr NAM Analysis and Obs verification Dec 24, 2006
1020 mb contour
Method
• Directly compare observations to interpolated model forecasts
• Emphasize East and West Coasts– Buoys eliminate terrain effects
• Emphasize Sea Level Pressure comparisons– SLP is good indicator of model performance (i.e., storm
strength and location can be defined and compared using SLP)
– Insufficient off-coast upper-level observations
Method
• Useful Tools for Comparing Forecast Skill
– Mean Absolute Error (MAE) – Mean Error, or Bias (ME)– Frequency of Errors which exceed an arbitrary “Large
Error” criterion (different for 24, 48, 72-hr forecasts)– Timelines of said “Large Errors” to cross-reference
failure dates between models– Error Histograms to compare distributions and sizes of
largest errors
•Matching Variance in Sea Level Pressure:
Eta to WRF-NAM operational switchover
> 7 mb
> 5 mb
> 3 mb
West Coast: Number of Large Errors by model and month
East Coast: Number of Large Errors by model and month
CMC - GEM major model update. Included: increase in vertical and horizontal resolution, new physics scheme, decreased time step, data assimilation changes
Number of Large Errors: West Coast *minus* East Coast
Mean Absolute Error: West Coast *minus* East Coast
For reference, typical MAE values: GFS West Coast average for winter 2006/07:
24-hr: 1mb; 48-hr: 1.4 mb; 72-hr: 2.0 mb
Comparing models: East Coast Mean Absolute Error *Standardized* to ECMWF
CMC – GEM update
Comparing models: West Coast Mean Absolute Error *Standardized* to ECMWF
Results• Comparing models: ECMWF generally outperforms
and NAM underperforms others. There are indications that a CMC-GEM model update significantly improved some forecasts, and that GFS has not performed as well this season as the past season.
• More “forecast bust” events occur on the West than East Coast for 24, 48, and 72 hour forecasts
• The East Coast is better forecast in terms of MAE in sea level pressure
Dates of Large Forecast Errors (by model)Errors greater than 5 mb
On the Horizon
On the Horizon
•Forecast Failures can be a result of initialization errors, insufficient “realism” of the model, and other model inadequacies. Knowing the most common causes of failure for high-impact forecast failures can help determine policy: do we invest more in improving models or in a new observation buoy?
•One cause of failure is High Model Sensitivity…
On the Horizon
Courtesy of Ryan Torn
On the Horizon
On the Horizon
On the Horizon
On the Horizon
On the Horizon
•Thirty-four of 420 forecasts showed the highest sensitivity
•Nineteen of the 34 “sensitive” cases (55%) resulted in multiple models meeting “large error criterion”
•Compare to: 10-20% of all forecasts result in multiple models meeting “large error criterion”
motivates a more formal investigation!
On the Horizon
Picture from fijaciones.org
Are Two Winters Representative? Currently adding data from 2004/05 and remainder of 2006/07 to further examine year-to-year variability in forecast skill
What About The Rest of the Country? And Beyond? Will examine open-ocean buoys, and land sites; over land will use 850 mb observations instead of SLP
What Are the Features of “Busts”: Forecast failures with high impacts on people still occur. What are their features? And why do some models fail when others don’t?
East Coast vs. West Coast: A Documentation of Model Forecast
Failures for Eta, NAM, GFS, GEM, and ECMWF
Garrett WedamLynn McMurdie, Cliff Mass
Thank You!