The International TIGGE Archive

Baudouin Raoult, Manuel Fuentes - ECMWFSteven Worley, Doug Schuster - NCAR

Bian Xiaofeng, Li Xiang - CMAMany Others!

3rd Thorpex Int. Science Symp., May 2009

Background: the TIGGE database

• Global ensemble forecasts to around 14 days generated routinely at different centres around the world

• Three archive centres: CMA, NCAR and ECMWF• Ten data providers:

– ECMWF, JMA (Japan), UK Met Office (UK), CMA (China), NCEP (USA), MSC (Canada), Météo-France (France), BOM (Australia), KMA (Korea), CPTEC (Brazil)

• Data exchanges using UNIDATA LDM, HTTP and FTP• 230 TB, 1.5 billion fields, growing by > 1 TB/week, 1.7 million fields/day

Background, Brief Time Line– March 2005, TIGGE Workshop at ECMWF

• Collect community input for TIGGE science and data requirements• Choose centralized archives for early start (ECMWF, NCAR, CMA)

– September 2005, meeting of technical experts at ECMWF• Representatives from ECMWF, NCAR, CMA, and NAEFS• Create technical implementation plan to address TIGGE needs

– November 2005, TIGGE Working Group at ECMWF• Present and get approval for the technical implementation plan

– October 2006, first EPS data is delivered– May 2006, User access to TIGGE archive starts– February 2008, all 10 international providers sending EPS

dataSignificant accomplishment, create 24x7 international

system in less than three years3rd Thorpex Int. Science Symp., May 2009

Quality assurance: homogeneity

• Homogeneity is paramount for TIGGE to succeed– The more consistent the archive the easier it will be to develop

applications

• There are three aspects to homogeneity:– Common terminology (parameter names, file names,…)– Common data format (format, units, …)– Definition of an agreed list of products (Parameters, Steps,

levels, …)

• What is not homogeneous:– Resolution– Base time (although most provider have a run at 12 UTC)– Forecast length– Number of ensemble

Strong governance

• Precise definition of:– Which products: list of parameters, levels, steps, units,…– Which format: GRIB2– Which transport protocol: UNIDATA’s LDM– Which naming convention: WMO file name convention

• Only exception: the grid and resolution– Choice of the data provider. Data provider to provide interpolation to

regular lat/lon– Best possible model output

• Many tools and examples:– Comprehensive documentation– Sample dataset available– Various GRIB2 tools, “tigge_check” validator, …– Scripts that implement exchange protocol

QA: Checking for homogeneity

• E.g. looking for wrong units?

Completeness

• The objective is to have 100% complete datasets at the Archive Centres

• Completeness may not be achieved for two reasons: – The transfer of the data to the Archive Centre fails– Operational activities at a data provider are interrupted

and back filling past runs is impractical

• Incomplete datasets are often very difficult to use• Most of the current tools used for ensemble

forecasts assume a fixed number of members from day to day

QA: Checking completeness (TIGGE history)

Archive Status and Monitoring, Variability between providers

Archive Status and Monitoring, Archive Completeness

PL = Pressure Level, PT = 320K θ Level, PV = ± 2 Potential Vorticity Level, SL = Single/Surface Level

Variable LvL ECWF UKMO JMA NCEP CMA CMC BOM MetF KMA CPTC

Geopotential Z PL

Specific H PL

T PL

U-velocity PL

V-velocity PL

Potential Vor PT

Potential T PV

U-velocity PV

V-Velocity PV

U 10m SL

V 10m SL

CAPE SL

Conv. Inhib. SL

Land-sea SL

Mean SLP SL

Orog. SL

Skin T SL

Snow D. H20 SL

Snow F. H20 SL

Archive Status and Monitoring, Archive Completeness

Variable LvL ECWF UKMO JMA NCEP CMA CMC BOM MetF KMA CPTC

Soil Moist. SL

Soil T SL

Sunshine D. SL

Surf. DPT SL

Surf. ATmax SL

Surf. ATmin SL

Surf. AT SL

Surf. P SL

LW Rad. Out SL

LH flux SL

Net Rad SL

Net Therm. Rad SL

Sensible Rad. SL

Cloud Cov SL

Column Water SL

Precipitation SL

Wilt. Point SL

Field Cap. SL

PL = Pressure Level, PT = 320K θ Level, PV = ± 2 Potential Vorticity Level, SL = Single/Surface Level

Access to TIGGE data

• Data is available to Research and Education after 48 hours– Self registration by agreeing to the terms and conditions

• Portals at CMA, NCAR and ECMWF

User Data Access and Support @ ECMWF

• TIGGE web site– Documentation– Model changes

• ECMWF portal offers:– Access to offline data– Aggregation along any axis (date, level, parameter, origin, ensemble,

…) – Provision of multi-model data on a single grid (regridding to any

lat/lon grid)– Sub-area selection– Reduces volumes to be downloaded by many order of magnitude

TIGGE @ ECMWF

TIGGE portal at ECMWF (using Ajax)

TIGGE Portal: grid selection

TIGGE Portal: area selection

TIGGE @ ECMWF

TIGGE @ ECMWF: Scheduling users’ requests

• Monitoring• Per user limits• Global limits

TIGGE @ ECMWF Batch access(recommended)

Perl or Python scripts

TIGGE@ECMWF: direct access to offline data (MARS)

User Data Access and Support, differences @ NCAR

• User data selection and subsetting equivalent to ECMWF

• 2.5 Weeks of Data Online• Archive organized in file groups by provider and

forecast initialization time– Pressure Level, Θ Level, PV Level, Single/Surface Level

• Multiple file download option – Direct through browser, using prepared ‘wget’ script

• Parameter download available through ‘curl’ scripts– Provides mechanism to build recurring requests

• Long-term file groups available by request

3rd Thorpex Int. Science Symp., May 2009

User Data Access and Support, differences @ NCAR

• Subset data format in GRIB2 or netCDF• Online queue to monitor subset data processing

progress• Subsetting completion email notification for users• TIGGE Research Support

– Library listing of software tools for data access and analysis– TIGGE Users Group web forum

• Community web space to share questions and findings• Provided by Unidata

• Archive of Tropical Cyclone (cxml) data for participating providers– Updated 6-hourly

3rd Thorpex Int. Science Symp., May 2009

User Data Access and Support, differences @ CMA

3rd Thorpex Int. Science Symp., May 2009

• Collaborated with ECMWF and CNIC

• Grid Middleware– Accessing the distributed

TIGGE data based on Grid middleware (GRIA, GOS) technologies

• Data retrieval and Derived Production Generation

• Request Monitoring and Query

• Access Control

User Data Access and Support, differences @ CMA

• Data Retrieval– Data Selection

• Selecting different data attributes, such as origin centre, date, base time, forecast time step, forecast type, level type, level and parameter.

• The different data options will be updated dynamically based on the user selection and on the constraints described in the data catalogue.

– Submit data request• If the number of requested fields is larger than the allowed limit, the

portal will suggest the users to split their data retrieval.

– Custom result products• Area selection• Grid conversion• Plotting

– Browse and download result products

User Data Access and Support, differences @ CMA

• Production Generation (e.g. ensemble mean)– Product selection

• Selecting the product and associated parameters based on the product catalogue• Submit product generation request

– In development, more products later

• Request Monitoring and Query– Check the real-time updates of the information about requests– Access the detailed monitoring information of requests for a period of time,

including request creation time, request finish time, process status, request condition, detailed processing information and result download.

– Query monitoring information according to the request date, request user id, request type and result status.

• Access Control– Several kinds of users (admin, data retrieval, product retrieval, …)

User Metrics at ECMWF

Active users

0

5

10

15

20

25

30

35

40

45

50

Dec-06

Jan-07

Feb-07

Mar-07

Apr-07

May-07

Jun-07

Jul-07

Aug-07

Sep-07

Oct-07

Nov-07

Dec-07

Jan-08

Feb-08

Mar-08

Apr-08

May-08

Jun-08

Jul-08

Aug-08

Sep-08

User Metrics at ECMWF : User Metrics at ECMWF

User Metrics at ECMWF : Data volumes

Data volumes (GBytes)

0

1000

2000

3000

4000

5000

6000

Dec-06

Jan-07

Feb-07

Mar-07

Apr-07

May-07

Jun-07

Jul-07

Aug-07

Sep-07

Oct-07

Nov-07

Dec-07

Jan-08

Feb-08

Mar-08

Apr-08

May-08

Jun-08

Jul-08

Aug-08

Sep-08

Retrieved

Delivered

Introduction to Poster Session demonstrationsInternational TIGGE Archive Centers

ECMWF http://tigge.ecmwf.int

NCAR http://tigge.ucar.edu

CMA http://wisportal.cma.gov.cn/tigge

Poster Sessions Demonstrations• When

• Tuesday, 15:00-16:30 (immediately following presentation)• Wednesday, 16:30-18:30• Any time, please ask Baudouin, Xiaofeng or Doug

• Purposes• Illustrate all features of the portals• Register users• Submit data requests• Demonstrate analysis tools

3rd Thorpex Int. Science Symp., May 2009

Credits

• Alex de Almeida Fernandes, Baudouin Raoult, Beth Ebert, Bian Xiaofeng, Dan Swank, Dave Stepaniak, David Ian Brown, Don Middleton, Dong-Joon Kim, Doug Schuster, Glenn Rutledge, Hannah Wilcox, Jean Nicolau, Jim Fraser, Joerg Urban, Julio Pablo Reyes Fernandez, Justin Cooke, Lang Hongliang, Lewis Poulin, Li Xiang, Ma Qiang, Manuel Fuentes, Michel Grenier, Okki Lee, Richard Dare, Richard Swinbank, Shintaro Yokoi, Simon Thompson, Steve Emmerson, Steve Worley, Steven Anthony, Tan Le, Tian Hao, Tom Yoksas, Waldenio Almeida, Yang Xin, Young-Youn Park, Yves Pelletier, …

• ... and many more