Bulk Statistics on Ensemble Model Forecasts for MDSS
Demo 2003
Paul Schultz
NOAA Forecast Systems Laboratory
June 17, 2003
The MDSS ensemble modeling component
• What is it?– Several computer model forecasts to supplement the NWS model
forecast services
• Why are we doing this?– Better forecasts. Just seeing if you’re paying attention.
• How does it work?– By combining multiple (imperfect) forecasts of the (imperfectly
observed) atmosphere, we can make a single ensemble forecast that is better than any of the forecasts that went into it.
2
Ensemble modeling
• Did it work during the 2003 Demo?– Not as well as it can. It shows promise. It can be improved.
3
The ensemble for Demo 2003
• Three models, two LBC source models, total of six ensemble members– models: MM5, RAMS, WRF– LBC sources (from NCEP): AVN, Eta– 6-hour cycle– 27-hour forecasts– 12-km grid
5
Bulk statisticsState variables, 12-hr forecasts
Feb 1 – Apr 8, 2003
6
Temperature (K) Wind speed (m/s) Dewpoint (K)
MM5-AVN 3.1 -0.7 2.5 +0.8 5.6 +1.5
MM5-Eta 3.0 -0.5 2.5 +0.8 5.5 +1.6
RAMS-AVN 5.8 -1.1 2.6 +1.6 6.5 -0.9
RAMS-Eta 5.9 -1.1 2.6 +1.7 6.9 -1.0
WRF-AVN 3.1 -0.4 2.4 +1.1 5.7 +1.4
WRF-Eta 3.1 -0.4 2.4 +1.0 5.7 +1.3
Precipitation verification
MDSS 0-3 h QPF Equitable Skill Score 133 runs from 1 Feb - 26 Mar 2003
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.01 0.1 0.25 0.5
Precip Threshold (in)
Eq
uit
able
Ski
ll S
core
MM5
RAMS
WRF
7
MDSS 0-3 h QPF Bias Score 133 runs from 1 Feb - 26 Mar 2003
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
0.01 0.1 0.25 0.5
Precip Threshold (in)
Fre
qu
ency
Bia
s
MM5
RAMS
WRF
A closer look
9 pm model runs, verifying only Iowa stations, entire expt
Improving the ensemble
• Remove unhelpful members– If we can’t fix RAMS problems, it’s gone
– Different LBC models don’t seem to help (?????)
8
Unhelpful members
MM5+Avn
WRF+Avn
MM5+Eta
WRF+Eta
The LBC models don’t add enough
diversity
Improving the ensemble
• Add good models– FSL/RUC a very good candidate for Demo 2004
• Change model configurations– WRF cloud/precip physics– Model cycle frequency, lead times, etc.
• Optimize use of available computing resources• Take advantage of what regional models do best
• Improve post-processing– Better PoP (probability of precip) estimates -- FSL– Better tuning procedures -- NCAR– Hope for “better” weather during tuning period
Reliability9
Percentage of expected FSL model runs
0
10
20
30
40
50
60
70
80
90
100
2/3 2/10 2/17 2/24 3/3 3/10 3/17 3/24 3/31 4/7
NCEP data problems Giant
snowstorm in Boulder
Planned power
outage at FSL
Reliability
• MM5 shows good reliability
• Others will improve with better scripting
10
Percentage of expected FSL model runs
0
10
20
30
40
50
60
70
80
90
100
MM5-AVN MM5-ETA RAMS-AVN RAMS-ETA WRF-AVN WRF-ETA
Photos from MDSS field trip
Bob Stradley and Ron Simmons
Downward-pointed radiometer mounted on rear-view mirror of
Jim Van Sickle’s truck
RWIS tower, I-35 south of Ames