Date post: | 14-Dec-2015 |
Category: |
Documents |
Upload: | kevin-goodness |
View: | 215 times |
Download: | 2 times |
A PERFORMANCE EVALUATION OF THE ETA - CMAQ
AIR QUALITY FORECAST MODEL
FOR THE SUMMER OF 2004
CMAS WorkshopChapel Hill, NC
20 October, 2004
Contributors
Brian Eder Daiwen KangTanya Otte Hsiu-mu Lin Jon Pleim Shaocai YuGeorge Pouliot STCKen SchereJeff YoungASMDARL, NOAA
All: On assignment to NERL, U.S. Environmental Protection Agency
Disclaimer: “The research presented here was performed under the Memorandum of Understanding between the U.S. Environmental Protection Agency (EPA) and the U.S. Department of Commerce's National Oceanic and Atmospheric Administration (NOAA) and under agreement number DW13921548. Although it has been reviewed by EPA and NOAA and approved for publication, it does not necessarily reflect their policies or views.
Forecast Configuration
- Eta Meteorology- CBIV Mechanism- SMOKE Emissions (Offline)- 12 km grid resolution - 22 Vertical Layers- 48 Hr. Forecast (12Z Init.)
Simulation Period
- 1 May – 30 September 2004- 1 June – 30 September 2004 (presented here)
Domain
Models-3 CMAQ
Lambert Conformal Projection
142
166
This performance evaluation used:
Hourly O3 (ppb) from EPA’s
AIRNOW network:
- over 600 stations (mostly urban)
- for a four month period (J, J, A, S)
A suite of statistical metrics for:
- discrete forecasts (MB, NMB, RMSE, NME)
- categorical forecasts (A, B, POD, FAR, CSI)
of hourly, max. 1-hr and max. 8-hr O3 (presented here).
Discrete Forecast / Evaluation
[Observed] [Observed] versus versus [Forecast] [Forecast]
Statistics
- Summary
- Biases MB
NMB
- Errors
N M EM odel O bs
O bs
N
N
1
1
100%( )
R M SE M odel O bsN
N
1 2
1
0 5
( )
.
Category Forecast / Evaluation
Observed Exceedances, Non-ExceedancesObserved Exceedances, Non-Exceedances versusversus
Forecast Exceedances, Non-ExceedancesForecast Exceedances, Non-Exceedances
a b
c d
Fo
reca
st E
xcee
danc
e
N
o
Yes
No YesObserved Exceedance
a b
c d
Ab c
a b c d%
100
Ba b
b d
FARa
a b%
100
C SIb
a b d
100%
PO Db
b d%
100
Monthly summaries:
- June, July, August, September
- Both Discrete and Categorical evaluation
Daily summaries:
- August 4 -12, Discrete evaluation
- Examine performance over a full synoptic cycle
- Illustrating both good and poor model performance
Forecast / Evaluation
Meteorological Conditions for the Summer (J, J, A) 2004
Meteorological Conditions for the Summer (J, J, A) 2004
Temperature Precipitation
Monthly Summaries
Obs
Mean
Model
Meanr
MB
(ppb)
NMB
(%)
RMSE
(ppb)
NME
(%)
46.1 53.9 0.51 7.8 16.9 14.1 24.0
A B POD FAR CSI a=40 b=19
99.5 0.79 25.3 67.8 16.5 c=18274 d=56
a b
c d
June Summary
Correlation Mean Bias (ppb)
NMB (%) NME (%)
June Summary
Domain Wide Daily Max. 8- Hour Means
0
10
20
30
40
50
60
70
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29
June
(pp
b)
OBS_MEAN MOD_MEAN
June Summary
1 15 30
June Summary
Bias
Error
1 15 30
Obs
Mean
Model
Meanr
MB
(ppb)
NMB
(%)
RMSE
(ppb)
NME
(%)
47.4 57.3 0.55 9.9 20.8 16.2 27.2
A B POD FAR CSI a=272 b=95
98.0 1.9 50.5 74.1 20.7 c=18258 d=93
July Summary
Obs
Mean
Model
Meanr
MB
(ppb)
NMB
(%)
RMSE
(ppb)
NME
(%)
43.4 55.1 0.62 11.7 27.1 16.5 30.9
A B POD FAR CSI a=232 b=19
98.7 5.2 39.6 92.4 6.8 c=19000 d=29
August Summary
Obs
Mean
Model
Meanr
MB
(ppb)
NMB
(%)
RMSE
(ppb)
NME
(%)
41.7 52.8 0.65 11.1 26.7 15.6 30.6
A B POD FAR CSI a=71 b=1
99.52 4.50 6.25 98.61 1.15 c=18018 d=15
September Summary
Daily Summaries
Synoptic Scale Meteorology 4 August
.01 0.5 1.0 2.0 3.0
20:15
Obs
Mean
Model
Meanr
MB
(ppb)
NMB
(%)
RMSE
(ppb)
NME
(%)
44.1
(58.5)
55.8
(62.1)
0.41
(0.27)
11.7
(3.5)
26.5
(6.0)
17.9
(13.1)
32.8
(17.3)
48 Hour Max. 8-Hour Ozone Forecast Init: 3 Aug (12 Z) Valid: 4 Aug TP (04 Z to 03 Z)
Model performance is poorer alongthe cold front where cloud cover andheavy rain dominate
Synoptic Scale Meteorology 5 August
.01 0.5 1.0 2.0 3.0
20:15
Obs
Mean
Model
Meanr
MB
(ppb)
NMB
(%)
RMSE
(ppb)
NME
(%)
43.8
(58.2)
51.6
(63.0)
0.73
(0.66)
7.8
(4.9)
17.9
(8.4)
12.3
(10.4)
22.3
(14.5)
48 Hour Max. 8-Hour Ozone Forecast Init: 4 Aug (12 Z) Valid: 5 Aug TP (04 Z to 03 Z)
Model performance is still poor alongthe cold front where cloud cover andheavy rain dominate
Synoptic Scale Meteorology 6 August
.01 0.5 1.0 2.0 3.0
20:15
Obs
Mean
Model
Meanr
MB
(ppb)
NMB
(%)
RMSE
(ppb)
NME
(%)
37.8
(52.6)
44.3
(55.3)
0.82
(0.67)
6.4
(2.6)
17.0
(5.0)
8.7
(6.9)
18.9
(10.1)
48 Hour Max. 8-Hour Ozone Forecast Init: 5 Aug (12 Z) Valid: 6 Aug TP (04 Z to 03 Z)
Model performance is very good throughoutthe domain, except for sections of NewEngland and southeast Ontario, where cloudcover is greatest.
Synoptic Scale Meteorology 7 August
.01 0.5 1.0 2.0 3.0
20:15
Obs
Mean
Model
Meanr
MB
(ppb)
NMB
(%)
RMSE
(ppb)
NME
(%)
41.2
(50.1)
45.9
(50.7)
0.79
(0.71)
4.7
(0.6)
11.5
(1.3)
8.6
(5.3)
16.2
(8.2)
48 Hour Max. 8-Hour Ozone Forecast Init: 6 Aug (12 Z) Valid: 7 Aug TP (04 Z to 03 Z)
Model performance is again very good, except for sections of New England, southeastOntario and Wisconsin, where cloud cover is greatest.
Synoptic Scale Meteorology 8 August
.01 0.5 1.0 2.0 3.0
20:15
Obs
Mean
Model
Meanr
MB
(ppb)
NMB
(%)
RMSE
(ppb)
NME
(%)
48.7
(54.8)
52.0
(55.2)
0.75
(0.62)
3.3
(0.4)
6.8
(0.7)
9.6
(6.6)
14.5
(9.5)
48 Hour Max. 8-Hour Ozone Forecast Init: 7 Aug (12 Z) Valid: 8 Aug TP (04 Z to 03 Z)
Model continues to perform well as thecP airmass undergoes modification. Warmertemperatures allow for higher ozone values (mean observed up to 48.7). New Englandstill has more cloud cover.
Synoptic Scale Meteorology 9 August
.01 0.5 1.0 2.0 3.0
20:15
Obs
Mean
Model
Meanr
MB
(ppb)
NMB
(%)
RMSE
(ppb)
NME
(%)
52.9
(57.1)
55.2
()58.1
0.78
(0.61)
2.3
(1.0)
4.4
(1.8)
8.0
(7.4)
11.6
(9.8)
48 Hour Max. 8-Hour Ozone Forecast Init: 8 Aug (12 Z) Valid: 9 Aug TP (04 Z to 03 Z)
Good model performance continues, as thecP airmass continues to warm. Highestozone values (52.9) observed on this day. Cloud cover in New England dissipates resulting in better model performance there.
Synoptic Scale Meteorology 10 August
.01 0.5 1.0 2.0 3.0
20:15
Obs
Mean
Model
Meanr
MB
(ppb)
NMB
(%)
RMSE
(ppb)
NME
(%)
49.4
(58.4)
57.7
(60.8)
0.69
(0.59)
8.6
(2.4)
17.4
(4.2)
15.6
(9.4)
24.5
(12.6)
48 Hour Max. 8-Hour Ozone Forecast Init: 9 Aug (12 Z) Valid: 10 Aug TP (04 Z to 03 Z)
Model performance degrades somewhat as cloud cover and precipitation increases overthe domain, most notably with a cold frontapproaching from the Great Lakes and a low forming over south
Synoptic Scale Meteorology 11 August
.01 0.5 1.0 2.0 3.0
20:15
Obs
Mean
Model
Meanr
MB
(ppb)
NMB
(%)
RMSE
(ppb)
NME
(%)
42.2
(51.8)
54.9
(60.7)
0.71
(0.33)
12.7
(9.0)
30.2
(17.3)
15.7
(12.7)
31.7
(19.3)
48 Hour Max. 8-Hour Ozone Forecast Init: 10 Aug (12 Z) Valid: 11 Aug TP (04 Z to 03 Z)
Model performance continues to suffer over the Great Lakes area as cloud cover, precipitation and very cool temperatures dominate this region.
Synoptic Scale Meteorology 12 August
.01 0.5 1.0 2.0 3.0
20:15
Obs
Mean
Model
Meanr
MB
(ppb)
NMB
(%)
RMSE
(ppb)
NME
(%)
32.5
(44.9)
51.1
(56.6)
0.38
(0.18)
18.6
(11.7)
57.4
(26.1)
21.0
(14.1)
58.0
(28.1)
48 Hour Max. 8-Hour Ozone Forecast Init: 11 Aug (12 Z) Valid: 12 Aug TP (04 Z to 03 Z)
Model performance at its worst as extensive cloud cover, precipitation and very cool temperatures dominate the entire domain.
Monthly Summary Obs
Mean
Model
Meanr
MB
(ppb)
NMB
(%)
RMSE
(ppb)
NME
(%)
June 46.1 53.9 0.51 7.8 16.9 14.1 24.0
July 47.4 57.3 0.55 9.9 20.8 16.2 27.2
August 43.4 55.1 0.62 11.7 27.1 16.5 30.9
September 41.7 52.8 0.65 11.1 26.7 15.6 30.6
Summary
Model performed reasonably well over the four month period, despiteanomalously cool and wet conditions.
Performance was closely tied to meteorology:
M very good performance when skies were generallyclear and void of precipitation,
M poorer when cloud cover and precipitation occurred.