Multi Model EnsemblesCTB Transition Project Team Report

Suranjana Saha, EMC (chair)Huug van den Dool, CPC

Arun Kumar, CPC

February 2007

TWO STUDIES WERE CONDUCTED USING THE CFS AND EUROPEAN DEMETER DATA TO EVALUATE THE FOLLOWING :

1.How extensive (long) should hindcasts be?

2.Does the NCEP CFS forecasts add to the skill of the European DEMETER-3 forecasts to produce a viable International Multi Model Ensemble (IMME) ?

How extensive (long) should hindcasts be?

Huug van den Dool Climate Prediction Center, NCEP/NWS/NOAA

Suranjana SahaEnvironmental Modeling Center, NCEP/NWS/NOAA

MODEL

SEC

CFS EC PLA METF UKM INGV LOD CERF MME8 (EW)ALLMODELS

MME3(EW) CFS+EC+UKM

SEC0 (NO SE)

2.1 1.2 0.0 0.0 0.0 0.4 0.2 0.0 0.2 0.9

SEC8(last 8 years)

4.3 7.1 1.4 1.4 7.5 1.4 0.4 2.2 3.8 8.6

SEC21(all 21 years)

11.2(0.33 cor)

8.0 0.4 0.4 8.6 0.6 0.1 0.5 2.0 17.0

Explained Variance (%) Feb 1981-2001; lead 3 (Nov starts); monthly T2m (US, CD data)

Explained Variance=Square of Anom Correlation

SEC : Systematic Error Correction; EW : Equal Weights

CFS=CFS, USA; EC=ECMWF; PLA=Max Planck Inst, Germany;

METF=MeteoFrance, France; UKM=UKMetOffice; INGV=INGV, Italy,

LOD=LODYC, France; CERF=CERFACS, France

Anomaly Correlation (%) Feb 1981-2001; lead 3 (Nov starts); monthly T2m (US, CD data)

WITH SEC8 WITH SEC21

SEC8-SEC21

SEC : Systematic Error Correction

Need more years to determine the SEC where/when the inter annual standard deviation is large

CONCLUSIONS• Without SEC (systematic error correction) there is no skill

by any method (for presumably the best month: Feb)

• With SEC (1st moment only), there is skill by only a few models (5 out of 8 are still useless)

• MME not good when quality of models varies too much

• MME3 works well, when using just three good models

CONCLUSIONS (contd)

• CFS improves the most from extensive hindcasts (21 years noticeably better than 8) and has the most skill. Other models have far less skill with all years included.

• Cross validation (CV) is problematic (leave 3 years out when doing 8 year based SEC?)

• Need more years to determine the SEC where/when the inter annual standard deviation is large

Skill in SST Anomaly Prediction for Nino-3.4[DJF 97/98 to AMJ 04]

5-member CFS reforecasts

50

60

70

80

90

100

1 2 3 4 5 6

Forecast Lead [Month]

Ano

mal

y C

orre

latio

n [%

]

CFSCMPCCACAMARCON

Skill in SST Anomaly Prediction for Nino-3.4 [DJF 97/98 to AMJ 04]

50

60

70

80

90

100

1 2 3 4 5 6

Forecast Lead [Month]

Ano

mal

y C

orre

latio

n [%

]

CFSCMPCCACAMRKCON

15-member CFS reforecasts

Skill in SST Anomaly Prediction for Nino-3.4[DJF 81/82 to AMJ 04]

5-member CFS reforecasts

50

60

70

80

90

100

1 2 3 4 5 6

Forecast Lead [Month]

Ano

mal

y C

orre

latio

n [%

]

CFS

CMP

CCA

CA

MAR

Skill in SST Anomaly Prediction for Nino-3.4 [DJF 81/82 to AMJ 04]

50

60

70

80

90

100

1 2 3 4 5 6

Forecast Lead [Month]

Ano

mal

y C

orre

latio

n [%

]CFSCMPCCACAMRK

15-member CFS reforecasts

Does the NCEP CFS add to the skill of

the European DEMETER-3 to produce a viable

International Multi Model Ensemble (IMME) ? Huug van den Dool

Climate Prediction Center, NCEP/NWS/NOAASuranjana Saha and Åke Johansson

Environmental Modeling Center, NCEP/NWS/NOAA

DATA USED

• DEMETER-3 (DEM3) = ECMWF + METFR + UKMO

• CFS

• IMME = DEM3 + CFS

• 1981 – 2001

• 4 Initial condition months : Feb, May, Aug and Nov

• Leads 1-5

• Monthly means

DATA/Definitions USED (contd)

• Anomaly Correlation (deterministic) and Brier Score (probabilistic)

• Ensemble Mean and PDF

• T2m and Prate

• Europe and United States

Verification Data :• T2m : Fan and van den Dool

• Prate : CMAP

“ NO consolidation, equal weights, NO Cross-validation “

BRIER SCORE FOR 3-CLASS SYSTEM

1. Calculate tercile boundaries from observations 1981-2001 (1982-2002 for longer leads) at each gridpoint.

2. Assign departures from model’s own climatology (based on 21 years, all members) to one of the three classes: Below (B), Normal (N) and Above (A), and find the fraction of forecasts (F) among all participating ensemble members for these classes denoted by FB, FN and FA respectively, such that FB+ FN+FA=1 .

3. Denoting Observations as O, we calculate a Brier Score (BS) as :BS={(FB-OB)**2 +(FN-ON)**2 + (FA-OA)**2}/3, aggregated over all years and all grid points.

{{For example, when the observation is in the B class, we have (1,0,0) for (OB, ON, OA) etc.}}

4. BS for random deterministic prediction: 0.444 BS for ‘always climatology’ (1/3rd,1/3rd,1/3rd) : 0.222

5. RPS: The same as Brier Score, but for cumulative distribution (no-skill=0.148)

Number of times IMME improves upon DEM-3 :

out of 20 cases (4 IC’s x 5 leads): Region EUROPE EUROPE USA USA

Variable T2m Prate T2m Prate

AnomalyCorrelation

9 14 14 14

Brier Score

16 18.5 19 20

RPS 14 15 19.5 20

“The bottom line”

Frequency of being the best model in 20 casesin terms of

Anomaly Correlation of the Ensemble Mean

“Another bottom line”

CFS ECMWF METFR UKMO

T2m USA 4 5 5 6

T2m EUROPE 3 5 6 5

Prate USA 7 3 3 6

Prate EUROPE 11 0 0 5

Frequency of being the best model in 20 casesin terms of

Brier Score of the PDF

“Another bottom line”

CFS ECMWF METFR UKMO

T2m USA 11 2 1 5

T2m EUROPE 10 3 1 3

Prate USA 17 0 0 2

Prate EUROPE 17 0 1 1

Frequency of being the best model in 20 casesin terms of

Ranked Probability Score (RPS) of the PDF

“Another bottom line”

CFS ECMWF METFR UKMO

T2m USA 9 4 1 6

T2m EUROPE 9 3 4 3

Prate USA 19 0 0 1

Prate EUROPE 18 0 0 1

CONCLUSIONS

• Overall, NCEP CFS contributes to the skill of IMME (relative to DEM3) for equal weights.

• This is especially so in terms of the probabilistic Brier Score

• And for Precipitation

• When the skill of a model is low, consolidation of forecasts (based on a-priori skill estimates) will reduce the chance that this model will be included in the IMME, and thus may lead to improvements in the skill of the IMME as obtained from equal weighting

CONCLUSIONS (Contd)

In comparison to ECMWF, METFR and UKMO, the CFS as an individual model does:• well in deterministic scoring (AC) for Prate• very well in probability scoring (BS) for Prate and

T2m over both domains of USA and EUROPE.

CONCLUSIONS (Contd)

• The weakness of the CFS is in the deterministic scoring (AC) for T2m (which is near average of the other models) over both EUROPE and USA

• While CFS contributes to IMME, it is questionable whether all other models contribute to CFS.

CONCLUSIONS (Contd)

• Skill (if any) over EUROPE or USA is very modest for any model, or any combination of models.

The AC for the ensemble mean gives a more “positive” impression about skill than the Brier Score, which rarely improved over climatological probabilities in this study

EUROPEAN IMMEUPDATE• RESULTS OF THIS STUDY WERE SENT TO THE ECMWF.

• THE DIRECTOR, ECMWF SHOWED INTEREST, BUT WANTED HIS OWN SCIENTISTS TO CARRY OUT THE EVALUATION.

• DR. DOBLAS-REYES (ECMWF) HAS DOWNLOADED THE CFS RETROSPECTIVE DATA FROM THE CFS SERVER AND IS IN THE PROCESS OF EVALUATING THE IMME, BUT USING THEIR LATEST EUROSIP DATA (INSTEAD OF THE DEMETER DATA).

RISKS• THE EUROPEANS MAY WELL WANT TO KEEP THEIR MME EUROPEAN.

• THEIR OPERATIONAL SEASONAL FORECAST PRODUCTS ARE NOT RELEASED IN REAL TIME (ONLY TO MEMBER STATES).

• BILATERAL AGREEMENTS MAY HAVE TO BE MADE TO OBTAIN THESE IN REAL TIME FOR ANY OPERATIONAL USE IN AN IMME WITH THE CFS.

OTHER COUNTRIES IN IMMEUPDATE

BMRC, AUSTRALIA• THE AUSTRALIANS ARE IN THE PROCESS OF COMPLETING THE RETROSPECTIVE FORECASTS WITH THEIR COUPLED MODEL.

• WHEN THESE FORECASTS ARE COMPLETED, A SIMILAR STUDY WILL BE CONDUCTED TO EVALUATE WHETHER THE AUSTRALIAN MODEL FORECASTS WILL BRING ADDITIONAL SKILL TO THE CFS FORECASTS.

BCC, BEIJING, CHINA• A SIMILAR SITUATION PERTAINS TO THE CHINESE METEOROLOGICAL AGENCY. WHEN THEY HAVE COMPLETED THE RETROSPECTIVE FORECASTS WITH THEIR COUPLED MODEL, WE WILL EVALUATE WHETHER THE CHINESE MODEL FORECASTS WILL BRING ADDITIONAL SKILL TO THE CFS FORECASTS.

NATIONAL MMEUPDATE• GFDL• HINDCAST DATA HAS BEEN OBTAINED FOR 4 INITIAL MONTHS (APR, MAY, OCT, NOV) FROM GFDL. THIS DATA IS BEING PROCESSED AND TRANSFERRED TO NCEP GRIDS FOR COMPARISON AND INCLUSION IN A MME WITH THE CFS.

• NASA• NOT READY TO START THEIR HINDCASTS

• NCARNOT READY TO START THEIR HINDCASTS.

BEN KIRTMAN (COLA) HAS DONE A FEW HINDCASTS WITH THE NCAR MODEL WHICH SHOW PROMISE. A FULL HINDCAST NEEDS TO BE DONE FOR EVALUATION IN A MME WITH THE CFS.