Post on 25-Jan-2016
description
transcript
A STATISTICAL EVALUATION OF TAMDAR DATA INSHORT-RANGE MESOSCALE NUMERICAL MODELS
Neil A. Jacobs1 and Yubao Liu2
1AirDat, LLC, Morrisville, NC 275602National Center for Atmospheric Research, Boulder, CO 80307
A STATISTICAL EVALUATION OF TAMDAR DATA INSHORT-RANGE MESOSCALE NUMERICAL MODELS
Neil A. Jacobs1 and Yubao Liu2
1AirDat, LLC, Morrisville, NC 275602National Center for Atmospheric Research, Boulder, CO 80307
Vertical Resolution Case Studies
Hypothesis: Increasing the number of model -levels in the lower to mid-troposphere will better utilize the greater observation density provided by TAMDARs, and result in a more accurate forecast.
Vertical Resolution Case Studies
Hypothesis: Increasing the number of model -levels in the lower to mid-troposphere will better utilize the greater observation density provided by TAMDARs, and result in a more accurate forecast.
Model Description:
RT-FDDA First-guess field
Little_R (MM5 Objective analysis) INTERPF MM5
Domains 1/2 (36-km/12-km)
Grell CP, MRF PBL, Mixed-phase (Reisner-1) microphysics
4 Simulations (96 h…was 14 h):
“TAMDAR” (36 -levels)
“Cntl” No TAMDAR (36 -levels…same as “TAMDAR”)
“TAMDAR+” (48 -levels: 6 1.5 km / all 12 5.5 km)
“Cntl+” No TAMDAR (48 -levels…same as “TAMDAR+”)
Initialized 1100 UTC 22 April 2005
Great Lakes late-season snow event (April 22-25, 2005)Great Lakes late-season snow event (April 22-25, 2005)
TAMDAR
TAMDAR+
Stage IV (4 km)
Cntl+
Cntl
1-h precipitation forecast (mm) and sea-level pressure (mb), as well as the 1-h Stage-IV analysis (mm), valid 1200 UTC 22 April 2005.
Cntl / Cntl+Scatter plot of 1-h QPF totals versus the 1-h Stage-IV analysis comparing matching grid point magnitudes (above 5 mm threshold) summed over each of the 14 forecast hours. The Cntl is blue, and the Cntl+ is red.
Cntl / Cntl+Scatter plot of 1-h QPF totals versus the 1-h Stage-IV analysis comparing matching grid point magnitudes (above 5 mm threshold) summed over each of the 14 forecast hours. The Cntl is blue, and the Cntl+ is red.
TAMDAR / TAMDAR+Scatter plot of 1-h QPF totals versus the 1-h Stage-IV analysis comparing matching grid point magnitudes (above 5 mm threshold) summed over each of the 14 forecast hours. The TAMDAR is blue, and the TAMDAR+ is red.
TAMDAR / TAMDAR+Scatter plot of 1-h QPF totals versus the 1-h Stage-IV analysis comparing matching grid point magnitudes (above 5 mm threshold) summed over each of the 14 forecast hours. The TAMDAR is blue, and the TAMDAR+ is red.
5
10
15
20
25
30
35
5 10 15 20 25 30 35
Matching grid point magnitude20050422 TAMDAR (Blue) and TAMDAR+ (Red)
Mo
de
l fo
reca
ste
d p
reci
p (
mm
)
Stage IV precip analysis (mm)
y = 9.21 + 0.127x R = 0.0869 (535)y = 3.44 + 0.621x R = 0.6732 (1102)
5
10
15
20
25
30
35
5 10 15 20 25 30 35
Matching grid point magnitude20050422 Cntl (Blue) and Cntl+ (Red)
Mo
de
l fo
reca
ste
d p
reci
p (
mm
)
Stage IV precip analysis (mm)
y = 8.79 + 0.196x R = 0.1765 (464)y = 7.05 + 0.325x R = 0.2346 (657)
2 f (x,y) 2 f (x,y)
x 2
2 f (x,y)
y 2
2[g(x,y) f (x,y)] [2g(x,y)] f (x,y) x 2 y 2 2 2
4e (x 2 y 2 )2 2
We can isolate cells based on magnitude, and retain only the precipitation associated with that cell
Laplacian edge detection:
Can also detect noise, so a smoothing Gaussian filter was tested…Laplacian of Gaussian:
•Both methods yield near-identical results
•All forecasts are regridded to 12-km
•An assumption is made that the closest cell (radial search) was the predicted cell.
•A weighted score was applied to the magnitude of the cell based on the linear distance (to the maximum) from “truth” (Stage-IV).
•For example, Stage-IV compared against itself would receive full weight. A score of 0 would mean either no cell was detected, or the distance was > 2(dm_cell+ds4_cell).
Precipitation cell isolation: who cares?A crude method to quantify QPF performance
Total
EXAMPLE: Raw Stage-IV 3-h accumulated precipitation data (Total), and the postprocessed (no minimum) isolated cells. The domain-2 data are mapped on the x-y grid.
5-mm
10-mm 15-mm
20-mm
Comparison of 12-h (4x3-h) QPF between TAMDAR+, TAMDAR, Cntl+, Cntl, and other various models regridded to the smallest grid (12-km), as well as the
Stage-IV analysis (“truth”) for 20-mm cells with 2-mm minimum bound.
Comparison of 12-h (4x3-h) QPF between TAMDAR+, TAMDAR, Cntl+, Cntl, and other various models regridded to the smallest grid (12-km), as well as the
Stage-IV analysis (“truth”) for 20-mm cells with 2-mm minimum bound.
The results presented here are consistent with preliminary findings from similar studies conducted at NCAR.
-100
0
100
200
300
400
500
600
700
0 10 20 30 40 50
20-mm cells (2-mm min) 12-h fcst sum
TAMDAR+TAMDARCntl+CntlRUCNAM218GFSStage IV
Precip (mm) per grid point
Wei
ghte
d gr
id p
oint
val
ue s
core
Results suggest that the addition of TAMDAR data in conjunction with increased vertical resolution improves the forecast skill for certain output parameters.
GFS, NAM, and RUC were included as “reference” models, and the improvement of the Cntl over these models is attributed to the 4DVAR ingestion technique of the RT-FDDA system.
However, the TAMDAR+ run shows significant improvements of 18-22% over the TAMDAR, Cntl, and the Cntl+ for this case.
This suggests that proper utilization of TAMDAR data plays a crucial role in forecast skill.
Results suggest that the addition of TAMDAR data in conjunction with increased vertical resolution improves the forecast skill for certain output parameters.
GFS, NAM, and RUC were included as “reference” models, and the improvement of the Cntl over these models is attributed to the 4DVAR ingestion technique of the RT-FDDA system.
However, the TAMDAR+ run shows significant improvements of 18-22% over the TAMDAR, Cntl, and the Cntl+ for this case.
This suggests that proper utilization of TAMDAR data plays a crucial role in forecast skill.
Model Description:
RT-FDDA First-guess field
Little_R (MM5 Objective analysis) INTERPF MM5
Domains 1/2 (36-km/12-km)
Grell CP, MRF PBL, Mixed-phase (Reisner-1) microphysics
4 Simulations (96 h…was 14 h):
“TAMDAR” (36 -levels)
“Cntl” No TAMDAR (36 -levels…same as “TAMDAR”)
“TAMDAR+” (48 -levels: 6 1.5 km / all 12 5.5 km)
“Cntl+” No TAMDAR (48 -levels…same as “TAMDAR+”)
Initialized 2300 UTC 29 August 2005
Hurricane Katrina (August 29, 2005)Hurricane Katrina (August 29, 2005)
TAMDAR+ Cntl+
Stage-IV
Sea-level pressure (mb) and 1-h precip. (in)Valid 0600 UTC 30 AUG 2005 (7-h Fcst)
Stage-IV
Cntl+TAMDAR+
Sea-level pressure (mb) and 1-h precip. (in)Valid 0900 UTC 30 AUG 2005 (10-h Fcst)
Stage-IV
Cntl+TAMDAR+
Sea-level pressure (mb) and 1-h precip. (in)Valid 1200 UTC 30 AUG 2005 (13-h Fcst)
Stage-IV
Cntl+TAMDAR+
Sea-level pressure (mb) and 1-h precip. (in)Valid 1500 UTC 30 AUG 2005 (16-h Fcst)
Stage-IV
Cntl+TAMDAR+
Sea-level pressure (mb) and 1-h precip. (in)Valid 1800 UTC 30 AUG 2005 (19-h Fcst)
Precipitation bands
TAMDAR+ is 4 mb deeper
850-hPa Relative Humidity (%)Valid 1800 UTC 30 AUG 2005 (19-h Fcst)
TAMDAR+ Cntl+
850-hPa Relative Humidity (%)Valid 0600 UTC 30 AUG 2005 (7-h Fcst)
Cntl+TAMDAR+
850-hPa RH Analysis DifferenceTAMDAR+ minus Cntl+2300 UTC 29 AUG 2005
850-hPa T Analysis DifferenceTAMDAR+ minus Cntl+2300 UTC 29 AUG 2005
From outer 36-km grid
Regions of RH responsible for future band formation
RAOB verification of RH band (case 2) is tough because of the space-time void.
TAMDAR was meant to fill this void, but verification against itself is a last choice.
Grell CP scheme trigger function is dependent on saturation (or near saturation) of moisture fields.
Minor differences in magnitude that exist near the CP scheme’s trigger threshold can tip the scales in a huge way hours later, which can be good or bad. Thus, proper assimilation of accurate data is key!
RAOB verification of RH band (case 2) is tough because of the space-time void.
TAMDAR was meant to fill this void, but verification against itself is a last choice.
Grell CP scheme trigger function is dependent on saturation (or near saturation) of moisture fields.
Minor differences in magnitude that exist near the CP scheme’s trigger threshold can tip the scales in a huge way hours later, which can be good or bad. Thus, proper assimilation of accurate data is key!
Why was the cyclone in TAMDAR+ 4 mb deeper when increased RH was the only difference seen in the analysis?
An increase in lower-tropospheric PV seen in the TAMDAR+ run appears to be linked to latent heat release from the precipitation bands around the cyclone (e.g., Bretherton 1966).
Preliminary findings suggest that the majority of geopotential height difference can be attributed to this additional PV.
Why was the cyclone in TAMDAR+ 4 mb deeper when increased RH was the only difference seen in the analysis?
An increase in lower-tropospheric PV seen in the TAMDAR+ run appears to be linked to latent heat release from the precipitation bands around the cyclone (e.g., Bretherton 1966).
Preliminary findings suggest that the majority of geopotential height difference can be attributed to this additional PV.
Precipitation Forecast Comparison
46 cases (22 March 2005- 4 June 2005)
AIRDAT (RT-FDDA-MM5 - TAMDAR)AIRNOT (RT-FDDA-MM5 - no TAMDAR)RUC, NAM, GFSStage-IV "truth"
Originally 49 cases, but 3 cases in May omitted based on initialization errors.
Precipitation Forecast Comparison
46 cases (22 March 2005- 4 June 2005)
AIRDAT (RT-FDDA-MM5 - TAMDAR)AIRNOT (RT-FDDA-MM5 - no TAMDAR)RUC, NAM, GFSStage-IV "truth"
Originally 49 cases, but 3 cases in May omitted based on initialization errors.
-1 104
0
1 104
2 104
3 104
4 104
5 104
6 104
0 10 20 30 40 50
5-mm cells (2-mm min) 12-h Fcst sum
AIRDAT TotalAIRNOT TotalGFS TotalNAM218 TotalRUC TotalStage4 Total
Wei
ghte
d gr
id p
oint
val
ue s
core
Precip (mm) per grid point
-1 104
0
1 104
2 104
3 104
4 104
5 104
0 10 20 30 40 50
10-mm cells (2-mm min) 12-h Fcst sum
AIRDAT TotalAIRNOT TotalGFS TotalNAM218 TotalRUC TotalStage4 Total
Wei
ghte
d gr
id p
oint
val
ue s
core
Precip (mm) per grid point
-5000
0
5000
1 104
1.5 104
2 104
2.5 104
3 104
3.5 104
0 10 20 30 40 50
15-mm cells (2-mm min) 12-h Fcst sum
AIRDAT TotalAIRNOT TotalGFS TotalNAM218 TotalRUC TotalStage4 Total
Wei
ghte
d gr
id p
oint
val
ue s
core
Precip (mm) per grid point
-5000
0
5000
1 104
1.5 104
2 104
2.5 104
3 104
0 10 20 30 40 50
20-mm cells (2-mm min) 12-h Fcst sum
AIRDAT TotalAIRNOT TotalGFS TotalNAM218 TotalRUC TotalStage4 Total
Wei
ghte
d gr
id p
oint
val
ue s
core
Precip (mm) per grid point
-5000
0
5000
1 104
1.5 104
2 104
2.5 104
0 10 20 30 40 50
25-mm cells (2-mm min) 12-h Fcst sum
AIRDAT TotalAIRNOT TotalGFS TotalNAM218 TotalRUC TotalStage4 Total
Wei
ghte
d gr
id p
oint
val
ue s
core
Precip (mm) per grid point
-5000
0
5000
1 104
1.5 104
2 104
0 10 20 30 40 50
30-mm cells (2-mm min) 12-h Fcst sum
AIRDAT TotalAIRNOT TotalGFS TotalNAM218 TotalRUC TotalStage4 Total
Wei
ghte
d gr
id p
oint
val
ue s
core
Precip (mm) per grid point
-2000
0
2000
4000
6000
8000
1 104
1.2 104
0 10 20 30 40 50
35-mm cells (2-mm min) 12-h Fcst sum
AIRDAT TotalAIRNOT TotalGFS TotalNAM218 TotalRUC TotalStage4 Total
Wei
ghte
d gr
id p
oint
val
ue s
core
Precip (mm) per grid point
-2000
0
2000
4000
6000
8000
0 10 20 30 40 50
40-mm cells (2-mm min) 12-h Fcst sum
AIRDAT TotalAIRNOT TotalGFS TotalNAM218 TotalRUC TotalStage4 Total
Wei
ghte
d gr
id p
oint
val
ue s
core
Precip (mm) per grid point
Object-oriented verification technique
Developed by Barbara Brown et al. (NCAR), and presented at previous GLFE.
All 49 cases…3 May outliers not removed
GFS not shown
0
50
100
150
200
0 2 4 6 8 10 12 14 16
AIRDATAIRNOTRUCNAMStage4
Nu
mb
ers
Intensity (mm) 12-h Fcst
NCAR verification 0.2 to 15-mm threshold
Improvement of QPF accuracy for short-range severe precipitation
8-11% - Control RT-FDDA-MM5 w/out TAMDAR (apples-to-apples)
34-55% - RUC w/out TAMDAR (apples-to-oranges)
71-84% - NAM (apples-to-squash)
>90% - GFS (apples-to-spaghetti)
A conservative estimate of potential improvement because…
• 4-month study utilized only 36 -levels• Stage-II/IV bias adjustment typically on low side (Smith and Krajewski 1991)
• Weighting/optimization of ingestion/parameterizations are still being refined
AirDat and NCAR findings are consistent despite different techniques
Improvement of QPF accuracy for short-range severe precipitation
8-11% - Control RT-FDDA-MM5 w/out TAMDAR (apples-to-apples)
34-55% - RUC w/out TAMDAR (apples-to-oranges)
71-84% - NAM (apples-to-squash)
>90% - GFS (apples-to-spaghetti)
A conservative estimate of potential improvement because…
• 4-month study utilized only 36 -levels• Stage-II/IV bias adjustment typically on low side (Smith and Krajewski 1991)
• Weighting/optimization of ingestion/parameterizations are still being refined
AirDat and NCAR findings are consistent despite different techniques
Cold-Start-MM5 Sensitivity Studies
Cumulus PBL NOTAMDAR TAMDAR
(TempError) (TempError)
Kain-Fritsch Blackadar 2.2C 2.0C
Grell Blackadar 2.3C 2.0C
Betts-Miller Blackadar 2.5C 2.4C
Kain-Fritsch2 Blackadar 2.2C 2.1C
Kain-Fritsch MRF 2.4C 2.3C
Grell MRF 2.5C 2.4C
Betts-Miller MRF 2.8C 2.6C
Kain-Fritsch2 MRF 2.5C 2.4C
Kain-Fritsch ETA-MY 2.8C 2.5C
Grell ETA-MY 2.6C 2.4C
Betts-Miller ETA-MY 2.9C 2.8C
Kain-Fritsch2 ETA-MY 2.9C 2.8C
(PrecipError) (PrecipError)
Kain-Fritsch Blackadar 0.45Ó 0.36Ó
Grell Blackadar 0.48Ó 0.42Ó
Betts-Miller Blackadar 0.64Ó 0.59Ó
Kain-Fritsch2 Blackadar 0.46Ó 0.38Ó
Kain-Fritsch MRF 0.52Ó 0.49Ó
Grell MRF 0.54Ó 0.50Ó
Betts-Miller MRF 0.69Ó 0.60Ó
Kain-Fritsch2 MRF 0.53Ó 0.51Ó
Kain-Fritsch ETA-MY 0.52Ó 0.47Ó
Grell ETA-MY 0.54Ó 0.49Ó
Betts-Miller ETA-MY 0.67Ó 0.59"
Kain-Fritsch2 ETA-MY 0.54Ó 0.48Ó
168 simulations on a CONUS 36-km grid
7 winter events (12 combinations)
Table: 144-h average of 3-h error
Error = | Forecast - ASOS* |
Blackadar good in winter despite 5-layer LSM
Snow cover / lack of veg. LSM’s influence
KF -> Grell may = feedback in warm-start
12/12 combinations: error with TAMDAR
*Automated Surface Observing System (NWS/FAA/DOD)
Cold-Start-MM5 Sensitivity Tests
0
0.5
1
1.5
2
2.5
3
12 24 36 48 60 72 84 96 108 120
2m-Temperature Error (C) CntlExp
Avg
. | T
emp.
Err
or |
(C)
Forecast Hour
2-m temp. error averaged for all 7 cases using KF/BlackadarCntl (No TAMDAR) Exp (TAMDAR)
This is a trend seen in all 7 cases! …not just an artifact of one “outlier”.
?• Better QPF > more accurate snow cover, albedo, and/or surface radiation > long- range surface temp. impact ?
• Better forecasted feedback from downstream “blocking” ?
• Weird Hovmoller teleconnection ?
• Lucky-7 ?
Objective was to obtain CPU speed benchmark for new 3GHz dual-core
Upcoming studies…
• Sampling Rate Impact Study 3 Parallel Simulations on Cold-Start MM5 36-km CONUS 12-km GLFE 48 -levels CNTL = No TAMDAR data EXP1 = TAMDAR data at “old” original sample rate EXP2 = TAMDAR data at “new” increased sample rate
• Variable Sampling Rate ? Based on forecasted dynamics
• Weighting Studies… Independent testing of ascent/descent …and independent testing of RH, T, winds Testing of radius and magnitude w.r.t. seasonal and diurnal variations
• QPF Verification: Round 2 Additional QPF verification studies, as well as other surface and upper-level verification will be needed after OSSE / weighting.
Upcoming studies…
• Sampling Rate Impact Study 3 Parallel Simulations on Cold-Start MM5 36-km CONUS 12-km GLFE 48 -levels CNTL = No TAMDAR data EXP1 = TAMDAR data at “old” original sample rate EXP2 = TAMDAR data at “new” increased sample rate
• Variable Sampling Rate ? Based on forecasted dynamics
• Weighting Studies… Independent testing of ascent/descent …and independent testing of RH, T, winds Testing of radius and magnitude w.r.t. seasonal and diurnal variations
• QPF Verification: Round 2 Additional QPF verification studies, as well as other surface and upper-level verification will be needed after OSSE / weighting.
Acknowledgments
Barbara Brown, Randy Bullock, and Wei Yu (NCAR’s object-based QPF verification)
Stan Benjamin and William Moninger (NOAA/ERL/FSL/GSD)
NCAR (OSSE computer support, etc.)
NASA Aeronautics Research Office’s Aviation Safety ProgramFAA Aviation Weather Research ProgramAirDat, LLC
Acknowledgments
Barbara Brown, Randy Bullock, and Wei Yu (NCAR’s object-based QPF verification)
Stan Benjamin and William Moninger (NOAA/ERL/FSL/GSD)
NCAR (OSSE computer support, etc.)
NASA Aeronautics Research Office’s Aviation Safety ProgramFAA Aviation Weather Research ProgramAirDat, LLC