Post on 27-Mar-2015
transcript
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
Systematic Errors
Thomas Jung
European Centre for Medium-Range Weather Forecasts
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
Scope of the Lecture
Question: “Do we still have significant systematic errors in the ECMWF forecasting system?”
If so, what are the main problems? How do systematic errors grow? How did systematic errors evolve over the years? How well do we simulate specific phenomena (e.g., blocking, extratropical cyclones)? Does increasing resolution help? Which techniques can be used to understand systematic error?
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
Introduction
Two principal sources of forecast error:
Uncertainties in the initial conditions (“observational error”)
Model error
Two kinds of forecast error
Random error (model+initial error)
Systematic error (model error*)
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
Concept of Systematic Error
)(ˆ)(ˆ),(ˆ ttttse odfdofd
Relatively straightforward to compute
BUT can be difficult to identify the origin of errors
Moreover there are pitfalls:• finite length (significance tests help partially)• apparent systematic error for short time series (loss of predictability) • observations might be biased
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
Loss of Predictability and “Systematic Error”
-6 -4 -2 0 2 4 6Temperature
0.00
0.01
0.02
0.03
0.04
0.05
0.06P
DF
Observed Anomaly“Systematic Error” when predictability
is lost
Fingerprint of loss of predictability: “Systematic error” resembles the opposite of the observed anomaly
“Observed”
“Forecast”
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
Example: Z500 DJF 2005/06
5120
5280
5280
5280
5440
5440
5440
5440
5600
56
00
5600
5600
5600
5760
5760
5760
5760
5760
5760
Mean Z500 Anomaly DJF 2006
-200
-140
-100
-80
-60
-40
-20
20
40
60
80
100
140
200
Observed Anomaly
5120
5280
5280
5280
5440
5440
5440
5440
5600
5600
5600
5600
5600
5760
5760
5760
5760
5760
5760
Mean Z500 Anomaly DJF 2006
-200
-140
-100
-80
-60
-40
-20
20
40
60
80
100
140
200
Mean Error @ D+10
Spatial correlation=-0.78
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
Uncertainty of our “Truth”
Precipitation GPCP (12-2 1990-2005)
-0.5
1
3
5
7
9
11
13
Precipitation ex51-GPCP (12-2 1990-2005)
-10
-4
-2
-0.5
0.5
2
4
10
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
Data
• Medium-range forecasts Primarily hindcasts from ERA-40 and ERA-Interim (robust
statistics).
• Seasonal integrations: 13 month long integrations started on 1st November of each of the
years 1962-2005 (or a subset of this period). Diagnosis of standard seasons DJF, MAM, JJA, SON (errors had at
least 1 month to grow, asymptotic errors). Most experiments at TL159 with 91 levels in the vertical.
Observed lower boundary conditions (uncoupled integrations).
• Observational data: ERA-40 Other “observational” data sets.
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
Systematic Error Growth
How do systematic errors grow throughout the forecast?
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
Systematic Z500 Error Growth from D+1 to D+10
520
540
560
560
580
580
580
(a) Z500 Difference D+1 FC-AN (DJF 1990-2001 ERA Interim)
-1-0.8-0.6-0.4-0.3-0.2-0.1
0.10.20.30.40.60.81
520
540
560
560
580
580
580
(b) Z500 Difference D+3 FC-AN (DJF 1990-2001 ERA Interim)
-3-2.4-1.8-1.2-0.9-0.6-0.3
0.30.60.91.21.82.43
520
540
560
560
580
580
580
(c) Z500 Difference D+5 FC-AN (DJF 1990-2001 ERA Interim)
-5-4-3-2-1.5-1-0.5
0.511.52345
520
540
560
560580
580
580
(d) Z500 Difference D+10 FC-AN (DJF 1990-2001 ERA Interim )
-10-8-6-4-3-2-1
12346810
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
Systematic Z500 Errors: Medium-Range and Beyond
-6-2 -2
-2
-2
-2
-2
-2-2
2
2
2
2
6
Z500 Difference 32R1-er40 (12-2 1963-2006)
-14
-12
-10
-8
-6
-4
-22
4
6
8
10
12
14
520
540
560
560
580
580
580
(a) Z500 Difference D+1 FC-AN (DJF 1990-2001 ERA Interim)
-1-0.8-0.6-0.4-0.3-0.2-0.1
0.10.20.30.40.60.81
520
540
560
560
580
580
580
(b) Z500 Difference D+3 FC-AN (DJF 1990-2001 ERA Interim)
-3-2.4-1.8-1.2-0.9-0.6-0.3
0.30.60.91.21.82.43
520
540
560
560
580
580
580
(c) Z500 Difference D+5 FC-AN (DJF 1990-2001 ERA Interim)
-5-4-3-2-1.5-1-0.5
0.511.52345
520
540
560
560
580
580
580
(d) Z500 Difference D+10 FC-AN (DJF 1990-2001 ERA Interim )
-10-8-6-4-3-2-1
12346810
D+10 ERA-Interim Asymptotic: 31R2
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
Evolution of Systematic Error
How did systematic errors evolve throughout the years?
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
Evolution of D+3 Systematic Z500 Errors
1983-1987 1993-1997 2003-2007
5120
5280
5280
5280
5440
5440
5440
5440
5600
5600
5600
56
00
5600
5760
5760
5760
5760
5760
5760
Mean Z500 Error DJF 1983-1987
-50
-35
-25
-20
-15
-10
-55
10
15
20
25
35
50
5120
5280
5280
5440
5440
5440
5440
5600
56
00
5600
5600
5600
5760
5760
5760
5760
57
60
5760
Mean Z500 Error DJF 1993-1997
-50
-35
-25
-20
-15
-10
-55
10
15
20
25
35
50
5120
5280
5280
5280
5440
5440
5440
5440
5600
5600
5600
56
00
5600
5760
576
0
5760
5760
57
60
5760
Mean Z500 Error DJF 2003-2007
-50
-35
-25
-20
-15
-10
-55
10
15
20
25
35
50
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
Evolution of Systematic Z500 Errors: Model Climate
-2
-2
-2
-2
-2
2
2
2
2
Z500 Difference 35R1-er40 (12-2 1963-2006)
-14
-12
-10
-8
-6
-4
-22
4
6
8
10
12
14
-2
-2
2
2
2
Z500 Difference 33R1-er40 (12-2 1963-2006)
-14
-12
-10
-8
-6
-4
-22
4
6
8
10
12
14
-2
-2
2
2
2
2
2
2
Z500 Difference 32R3-er40 (12-2 1963-2006)
-14
-12
-10
-8
-6
-4
-22
4
6
8
10
12
14
-6
-6
-6
-6
-2
-2
-2
-2
-2
-2
-2
-2
-2
2
2
2
2
2
2
6
6
Z500 Difference 32R2-er40 (12-2 1963-2006)
-14
-12
-10
-8
-6
-4
-22
4
6
8
10
12
14
-6
-2 -2
-2
-2
-2
-2
-2-2
2
2
2
2
6
Z500 Difference 32R1-er40 (12-2 1963-2006)
-14
-12
-10
-8
-6
-4
-22
4
6
8
10
12
14
-10
-6
-6
-6
-6
-2
-2
-2
-2
-2
-2
-2
-2
2
2
22
2
2
Z500 Difference 31R1-er40 (12-2 1963-2006)
-14
-12
-10
-8
-6
-4
-22
4
6
8
10
12
14
-6
-6
-6
-6
-6
-2
-2
-2
-2
-2
-2
-2
-2
2
2
2
2
6
Z500 Difference 30R1-er40 (12-2 1963-2006)
-14
-12
-10
-8
-6
-4
-22
4
6
8
10
12
14
-6
-6
-6
-6
-6
-2
-2
-2-2
-2
-2
-2
-2
-2
2
2
2
2
2
6
Z500 Difference 29R2-er40 (12-2 1963-2006)
-14
-12
-10
-8
-6
-4
-22
4
6
8
10
12
14
35R1 33R1 32R3 32R2
32R1 31R1 30R1 29R2
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
Systematic Z500 Errors: Impact of Recent Changes
-2
-2
-2
-2
-2
2
22
2
2
2
Z500 Difference f127-er40 (12-2 1990-2005)
-14
-12
-10
-8
-6
-4
-22
4
6
8
10
12
14
-6
-2
-2-2
-2
-2
-2
2
2
2
2
2
2
6
Z500 Difference f3wt-er40 (12-2 1990-2005)
-14
-12
-10
-8
-6
-4
-22
4
6
8
10
12
14-2
-2
-2
2
2
2
2
2
2
2
2
2
Z500 Difference f251-er40 (12-2 1990-2005)
-14
-12
-10
-8
-6
-4
-22
4
6
8
10
12
14
-2
-2
-2
2
2
2
2
2
2
2
2
Z500 Difference f3y0-er40 (12-2 1990-2005)
-14
-12
-10
-8
-6
-4
-22
4
6
8
10
12
14
-2
-2
-2
-2
-2
-2
-2
2
2
Z500 Difference f3y1-er40 (12-2 1990-2005)
-14
-12
-10
-8
-6
-4
-22
4
6
8
10
12
14
-2
-2
-2
-2
-2
-2-2
2
2
2
Z500 Difference f3y2-er40 (12-2 1990-2005)
-14
-12
-10
-8
-6
-4
-22
4
6
8
10
12
14
Control Old Convection Old TOFD
Old Vertical Diff Old Radiation Old Soil Hydrology
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
Phenomena
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
510
510
510
510
520
520
520
520
520
530
530
530
530
540
540
540
540
550
550
550
550
560
560
560
560
560
570
570
570
570
570
570
570
580
580
ECMWF Analysis VT:Saturday 23 December 2006 12UTC 500hPa Geopotential
Blocking Methodology
L
H
L
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
-180 -120 -60 0 60 120 180Latitude
0
10
20
30
40
Blo
ckin
g F
req
uenc
y (%
)
Blocking Frequency Errors (23r4)
DJF 1990-2001
ERA-40D+1D+4D+7D+10
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
Blocking Frequency Errors (31R2)
DJF 1990-2001
ERA-40D+1D+4D+7D+10
-180 -120 -60 0 60 120 180Latitude
0
10
20
30
40
Blo
ckin
g F
req
ue
ncy
(%
)
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
Blocking Frequencies: DJF 1990-2005
-180 -120 -60 0 60 120 180Longitude
0
10
20
30
40B
lock
ing
Fre
qu
en
cy (
%)
ERA4033R132R329R2-32R2
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
New Blocking Diagnostic
• Feature tracking software from Kevin Hodges
• Track positive stream function anomalies at 1000 hPa
• Some filtering: T3-T12
• Diagnostics: Number, frequency, lifetime, …
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
Frequency of Z1000 Maxima (er40 DJFM 1962-2005)
1
2
3
4
5
6
7
8
9
10
12
15
20
25
30
Frequency of Z1000 Maxima (f18l-er40 DJFM 1962-2005)
-7.5
-5
-4
-3
-2
-1
-0.5
0.5
1
2
3
4
5
7.5
Frequency of Z1000 Maxima (f127-er40 DJFM 1962-2005)
-7.5
-5
-4
-3
-2
-1
-0.5
0.5
1
2
3
4
5
7.5
Frequency of Z1000 Maxima (f127-f18l DJFM 1962-2005)
-7.5
-5
-4
-3
-2
-1
-0.5
0.5
1
2
3
4
5
7.5
Frequency of Anti-cyclones (1962-2005 DJF)
ERA40
33R1-ERA40
31R1-ERA40
33R1-31R1
Long-lived anticyclones only
(> 2 days)
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
Average Lifetime Z1000 Maxima (er40 DJFM 1962-2005)
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
6
7
8
9
10
Average Lifetime Z1000 Maxima (f18l-er40 DJFM 1962-2005)
-6
-4.5
-3.5
-3
-1.5
-1
-0.5
0.5
1
1.5
2.5
3.5
4.5
6
Average Lifetime Z1000 Maxima (f127-er40 DJFM 1962-2005)
-6
-4.5
-3.5
-3
-1.5
-1
-0.5
0.5
1
1.5
2.5
3.5
4.5
6
Average Lifetime Z1000 Maxima (f127-f18l DJFM 1962-2005)
-6
-4.5
-3.5
-3
-1.5
-1
-0.5
0.5
1
1.5
2.5
3.5
4.5
6
Average Lifetime of Anticyclones (1962-2005 DJF)
ERA40
33R1-ERA40
31R1-ERA40
33R1-31R1
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
Extratropical Synoptic Systems
Two appoaches:
• Compute standard deviation of highpass or bandpass filtered time series (“Eulerian” approach).
• Track individual systems (“Lagrangian” approach).
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
“Observed” Tracks of Long-lived Cyclones
ETC ECMWF Analysis (DJFM 05/06)
940 950 960 970 980 990 1000 1010 1020
n= 515
>2 days
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
Sensitivity to Horizontal Resolution: 29R1
Wintertime Cyclones over the Northern Hemisphere
1980 1985 1990 1995 2000 2005Time
100
150
200
250
300
350
400N
um
be
r o
f Cyc
lon
es
ERA-40EC-T95EC-T159EC-T255
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
Sensitivity to Horizontal Resolution: 31R1
Wintertime Cyclones over the Northern Hemisphere
1990 1995 2000 2005Time
200
300
400
500
600
700
800
Nu
mb
er
of C
yclo
ne
s
ERA-40T95T159T255T511
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
Number of Extratropical Cyclones (DJFM)
Lifetime > 1day
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
Computational Effort
• 12% of all CPUs on HPCE cluster
• Wall clock time about 20 hours• About 70 times more expensive than TL95!
1 Integration (151 days) @ TL511L91:
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
Synoptic Activity Error: Evolution
2222
22 2 2
44
4
4
4444
44 4 4
6
6 66
6666
66
6
6
6
8
8 88
88
8
8
8
8
8
8
8
8
10
10
10 1010
10
10
10
10
10
1212
12
12
Synoptic Z500 Activity: Difference 33R1-ERA40 (12-2 1962-2005)
-4-3-2.5-2-1.5-1-0.50.511.522.534
2222
22 2 2
44
4
4
4444
44 4 4
6
6 66
6666
66
6
6
6
8
8 88
88
8
8
8
8
8
8
8
8
10
10
10 1010
10
10
10
10
10
1212
12
12
Synoptic Z500 Activity: Difference 32R3-ERA40 (12-2 1962-2005)
-4-3-2.5-2-1.5-1-0.50.511.522.534
2222
22 2 2
44
4
4
4444
44 4 4
6
6 66
6666
66
6
6
6
8
8 88
88
8
8
8
8
8
8
8
8
10
10
10 1010
10
10
10
10
10
1212
12
12
Synoptic Z500 Activity: Difference 32R2-ERA40 (12-2 1962-2005)
-4-3-2.5-2-1.5-1-0.50.511.522.534
2222
22 2 2
44
4
4
4444
44 4 4
6
6 66
6666
66
6
6
6
8
8 88
88
8
8
8
8
8
8
8
8
10
10
10 1010
10
10
10
10
10
1212
12
12
Synoptic Z500 Activity: Difference 31R1-ERA40 (12-2 1962-2005)
-4-3-2.5-2-1.5-1-0.50.511.522.534
2222
22 2 2
44
4
4
4444
44 4 4
6
6 66
6666
66
6
6
6
8
8 88
88
8
8
8
8
8
8
8
8
10
10
10 1010
10
10
10
10
10
1212
12
12
Synoptic Z500 Activity: Difference 30R1-ERA40 (12-2 1962-2005)
-4-3-2.5-2-1.5-1-0.50.511.522.534
2222
22 2 2
44
4
4
4444
44 4 4
6
66
6
6666
66
6
6
6
8
8 88
88
8
8
8
8
8
8
8
8
10
10
10 1010
10
10
10
10
10
1212
12
12
Synoptic Z500 Activity: Difference 35R1-er40 (12-2 1963-2006)
-4-3-2.5-2-1.5-1-0.50.511.522.534
35R1
32R3
31R1
32R2
30R1
33R1
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
The Madden-and Julian Oscillation
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
Schematic of the MJO
From Madden and Julian (1994)
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
Near Global Impact of the MJO (Precipitation)
Conv.: Indian Ocean Conv.: Maritime Continent
Conv.: Central Pacific Conv.: WH/Africa
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
The Madden and Julian Oscillation
ERA-40
ECMWF Model
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
Conclusions (1)
Main systematic errors: Concept of systematic error is very straightforward.
But there are pitfalls:
• Short time series (sampling issues + loss predictability).
• Uncertainty about the true state of the atmosphere.
Systematic error are a clear sign of model error.
Understanding systematic error is challenging.
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
Conclusions (2)
Main systematic errors:Do we still have systematic errors in the ECWMF model?
• Yes, we do.
• Most systematic errors have been substantially reduced in recent years.
• Some key-challenges remain, however!
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
Conclusions (3)
Main systematic errors:
Tropical circulation + hydrological cycle
Madden-Julian Oscillation
Indian Summer Monsoon
Quasi-Biennial Oscillation
Euro-Atlantic blocking
Synoptic activity (severe wind storms)
Others (surface related fields)
Key-challenges:
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
Further Reading
Bechtold, P., M Koehler, T. Jung, M. Leutbecher, M. Rodwell and F. Vitart, 2008: Advances in simulating atmospheric variability with the ECMWF model: From synoptic to decadal time scales. Quart. J. Roy. Meteor. Soc., 134, 1337-1351.
Jung, T. and A.M. Tompkins, 2003: Systematic Errors in the ECMWF Forecasting System. ECMWF Technicial Memorandum 422. http://www.ecmwf.int/publications/library/do/references/list/14
Jung, T., A.M. Tompkins, and R.J. Rodwell, 2004: Systematic Errors in the ECMWF Forecasting System. ECMWF Newsletter, 100, 14-24.
Jung, A.M. Tompkins, and R.J. Rodwell, 2005: Some Aspects of Systematic Errors in the ECMWF Model. Atmos. Sci. Lett., 6, 133-139.
Jung, T. and co-authors, 2009: The ECMWF model climate: Recent progress through improved physical parametrizations. ECMWF Seminar Proceedings on Parameterization of Subgrid-scale Processes, 233-249. Available from the ECMWF website.
Jung, T., 2005: Systematic Errors of the Atmospheric Circulation in the ECMWF Model. Quart. J. Roy. Meteor. Soc., 131, 1045-1073.
Jung, T., S.K. Gulev, I. Rudeva and V. Soloviov, 2006: Sensitivity of extratropical cyclone characteristics to horizontal resolution in the ECWMF model. Quart J. Roy. Meteor. Soc., 132, 1839-1857.
Hoskins, B.J. and K.I. Hodges, 2002: New Perspectives on the Northern Winter Storm Tracks. J. Atmos. Sci., 59, 1041-1061.
Koehler, M., 2005: ECMWF Technicial Memorandum 422. http://www.ecmwf.int/publications/library/do/references/list/14
Palmer, T.N., G. Shutts, R. Hagedorn, F. Doblas-Reyes, T. Jung, and M. Leutbecher, 2005: Representing Model Uncertainty in Weather and Climate Prediction. Ann. Rev. Earth. Planet Sci., 33, 163-193.
Rodwell, MJ and T Jung, 2008: Understanding the local and global impacts of model physics changes: An aerosol example. Quart. J. Roy. Meteor. Soc., 134, 1479-1497.
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
Recent Model Changes
29R2 28 Jun 2005 Modifications to convection
30R1 1 Feb 2006 Increased resolution (L60 to L91)
31R1 12 Sep 2006 Revised cloud scheme (ice supersaturation + numerics); implicit computation of convective transports; introduction of orographic form drag scheme; revised GWD scheme
32R1 not operational
New short-wave radiation scheme; introduction of McICA cloud-radiation interaction; MODIS aerosol; revised GWD scheme; retuned ice particle size
32R2 5 Jun 2007 Minor changes to forecast model
32R3 6 Nov 2007 New formulation of convective entrainment and relaxation time scale; reduced vertical diffusion in the free atmosphere; modification to GWD scheme (top of the model); new soil hydrology scheme
33R1 3 Jun 2008 Slightly increased vertical diffusion; increased orographic form drag; retuned entrainment in convection scheme; bugfix scaling of freezing term in convection scheme; changes to the surface model
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
Sensitivity to Model Formulation
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
Systematic Errors: AMIP Models
Training Course Presentation, 16th March 2009 – Thomas Jung: Systematic Model Errors.
Demeter Models (DJF Precipitation)
CERFACS LODYC
MPIMetOfficeMeteo-France
ECMWF