Working Group/Observational Data

Spring 2015

Co-Chairs:Mr. Dennis Hobson, ACC/A5WMr. Kyle Rushing, NAVO

Exec. Secretary: Mr. Anthony Ramirez,

OFCM

2

Overview

Scope of Responsibility Members Ongoing Activities Action Items Upcoming Activities

3

Scope of Responsibility(Modified extract from Terms of Reference) Facilitates the acquisition, processing, and exchange of observational data among the National Operational Processing Centers (OPCs) and other related data centers and supports the Committee for Operational Processing Centers (COPC) through its executive agent, the Working Group for Cooperative Support and Backup (WG/CSAB). The primary focus areas:

• Facilitate the acquisition, processing and exchange of meteorological, oceanographic, and space environmental data

• Provide an interface between the OPCs, their research and development partners in the Joint Center for Satellite Data Assimilation (JCSDA), and the other national data and prediction centers for the purpose of coordinating and satisfying national requirements for observational data

• Coordinate data formatting standards where practical and the implementation of approved data product enhancements and new data products

• Coordinate observational data issues that overlap related responsibilities of other OFCM committees and COPC groups

• Ensure the OPCs and related data centers are provided the maximum quality and optimum quantity of observational environmental data streams required for assimilation into their respective processes

4

Member Agencies/Entities

NOAA• National Environmental Satellite, Data, and Information Service/Office of

Satellite and Product Operations (NESDIS/OSPO), Suitland, MD• National Centers for Environmental Information (NCEI)

• Center for Weather and Climate (CWC) (formerly NCDC)

• National Weather Service (NWS), Silver Spring, MD• National Centers for Environmental Prediction (NCEP), College Park,

MDo Space Weather Prediction Center (SWPC)o NCEP Central Operations (NCO-Silver Spring) (formerly NWS/Telecommunications Operations Center (TOC))o Environmental Modeling Center (EMC)

• National Data Buoy Center (NDBC)

• Office of Oceanic and Atmospheric Research (OAR)

5

Member Agencies/EntitiesAir Force

• HQ AF/A3W, Pentagon, Washington DC• HQ Air Combat Command (ACC/A5W OL-P), Offutt AFB, NE • 557th Weather Wing, Offutt AFB, NE

• 14th Weather Squadron, Asheville, NC

Navy• Fleet Numerical Meteorology and Oceanography Center (FNMOC),

Monterey, CA.• Naval Oceanographic Office (NAVOCEANO), Stennis Space Center, MS• Naval Research Laboratory-Marine Meteorology Division

Joint Center for Satellite Data Assimilation (JCSDA) (Headquarters), College Park, MD

6

WG/Observational Data  

Co-Chairs Mr. Dennis Hobson, ACC/A5W Mr. Kyle Rushing, NAVO

NCEP Mr. Jeff Ator - EMC Mr. Walter Smith – NCO/SS

Air Force Mr. Eric Wise Mr. Dennis Hobson Mr. Al Zamiska

14th WS Mr. Randy Haeberle

NESDIS/OSPO Mr. Vincent Tabor 

FNMOC Mr. James Vermeulen Dr. Justin Reeves -- new hire Dr. Patricia Pauley - NRL

NAVO   Mr. Kyle Rushing   Mr. Keith Willis

JCSDA Dr. James Yoe

NWS/NDBC Mr. Rex Hervey

7

 

NOAA/OAR Dr. Stephen Piotrowicz

NOAA/IOOS Mr. Derrick Snowden

NESDIS/NCEI/CWC (formerly NCDC) Vacant (seeking replacement)

NWS/SWPC Dr. Rodney Viereck

OFCM Mr. Anthony Ramirez (Exec. Secretary)

WG/Observational Data

8

MET-7 Follow-on EUMETSAT Council publicly has stated they do not

intend to support IO imaging beyond end of CY 2016 and planned MET-7 end of life/deorbit

Mitigation to gap in geostationary imaging of IO region HQ AF directive is to not exploit China/Russia

METSAT COMS-1 posn 12 deg further west (128 deg E) than

H-8 (140 deg E) Explore quality of INSAT 3-D data received by

NOAA on devel sys Potential COA if favorable result: pursue

operational feed? Lack of geo imagery will force reliance on polar-

orbiting platforms for DoD forecaster application

Ongoing Activities

9

Ongoing Activities

Acquisition of INSAT-3D data

India National Satellite - 3D is a multi-purpose geostationary satellite with the potential to extend high temporal-resolution geostationary coverage to the Indian Ocean/Arabian Sea region

Acquisition effort ongoing (WG/CSAB update in upcoming brief)

10

Ongoing Activities AF/Navy Interest in Operational Feed

of NASA Global Precipitation Measurement

(GPM) Data GPM Spacecraft launched Feb 27, 2014 AF submitted SDPR Mar 2014 to acquire Level 1b GPM

brightness temps and radar enhanced precipitation retrievals; Navy added to SDPR

AF using in Climate Prediction Cen Morphing Technique (CMORPH) Replaced TRMM in CMORPH process

Navy will use in NAVDAS-AR and tactical imagery applications

NOAA partners in JCSDA have access to some GPM/GMI Q3 FY14: GMI L1B, L1C, and L1C-R data had been

BUFR-ized Q4 FY14 GMI L1C-R data ingested into GSI, and

optimization of GMI for data assimilation is beginning (QC, obs error, bias)

Status: NESDIS accelerated CCB process, made GPM available operationally 2 April; TRMM data ceased 9 April AF CMORPH production uninterrupted

11

Ongoing Activities

NAVO interest in acquiring Oceansat II

data from NOAA for Ocean Color (OCM sensor -- Ocean Color Monitor)

NOAA states that the OCM-2 data will not be global Data covers Indian waters only The onboard recorders are not configured to record

globally

NAVO to recommend NRL skip incorporation of OCM-2 into AOPS

Closed

12

Data Requestor Provider Status

GOES-13/15 Level 1c Sounder

AFWA NESDIS Closed

GCOM-W Lvl 1b and Lvl 2 Prods

AFWA/FNMOC NESDIS Open

NASA GPM GMI Lvl 1b Brightness temps

AFWA/FNMOC NESDIS Closed

NASA GPM GMI Lvl 2 Radar Enhanced Precip Retrievals

AFWA/FNMOC NESDIS Closed

MODIS Active Fire Prod AFWA NESDIS Open

Wildfire Auto Biomass Algorithm (WF-ABBA)

AFWA/FNMOC NESDIS Open

VIIRS Polar Winds FNMOC NESDIS Closed

Satellite Data Requests

13

Data Requestor Provider Status

NUCAPS Products FNMOC NESDIS Open

1 KM IMS NAVO NESDIS Open

SMAP AFWA NESDIS Open

F-19 EDR, SDR, TDR and UPP Products

NESDIS FNMOC Closed

Satellite Data Requests

14

.

Radar Data Exchange

FNMOC interested in acquiring NEXRAD Level II files direct from NOAA vs 557th WW.

557th WW acquires NEXRAD Level II and III files from NCO SS server using Local Data Manager (LDM); rehosts on 557th WW server for distribution to mil users which employ Gibson Ridge software.

Initially, FNMOC was not able to reliably access data via 557th WW server; 557th WW identified problem and implemented fix. NEXRAD radar data feed between FNMOC & 557th WW has been rock-solid and NEXRAD data is being assimilated into FNMOC’s operational COAMPS model.

Reliability/stability of NEXRAD feed from 557th WW has provided much appreciated “breathing room”, giving IA/IT personnel time to properly engineer a server/storage solution of NEXRAD Level II data at NCO SS.

FNMOC & NCO SS currently addressing LDM IA concerns and in the process of establishing necessary network protocols via OTN.

Ongoing Activities

15

.

Radar Data for Guantanamo Bay Area

NCEP/NCO Silver Spring, on behalf of NWS Office of Dissemination, requested radar data for Guantanamo Bay area

Gap in the radar mosaic for that area

Discoveries: FAA advised that ARSR4 radar at GITMO has a weather

channel ARSR4 is FAA ‘owned’ but DOD/DHS funded to keep the

systems operational  Not Nexrad quality but may be sufficient to cover any

gaps in the mosaic

Status: NWS/Radar Operations Center coordinating with

NCEP/NCO SS to acquire and provide this data stream

Action Items

16

.

Environmental Satellite Data Annex update

Background: Annex slightly outdated (5-yr cost projections), GC comments from DAPE MOA, Transition from AFWA IDPS

Draft ESDA developed and reviewed by the METSAT members of the WG/OD; submitted to WG/CSAB for the next level of coord

Several parts of the document (primarily updated funding projections) are yet to be finalized.

WG/CSAB update follows this presentation

Action Items

17

.

Monitor/coordinate resolution of issues that arise due to WMO text to

BUFR transition BACKGROUND:

WMO is migrating from Traditional Alphanumeric Code (TAC) forms to Binary Universal Form for the Representation of meteorological information (BUFR)

OPCs have expected a complicated migration and have been responding to issues as they arise.

TAC distribution was SCHEDULED to end in November 2014, but…

Most (but not all) countries will continue to distribute in BOTH TAC and BUFR for months or years

TAC data counts will diminish as countries discontinue TAC distribution

If production centers do not have replacement BUFR products, we could experience negative impacts

OPCs need to temporarily use BOTH TAC and BUFR data WMO will no longer approve any changes to or additions

of TAC code

Action Items

18

.

Several items of concern as the global community transitions to BUFR

Regional Telecommunication Hub (RTH) Washington is not in receipt of a significant number of BUFR bulletins from other RTH sites

Some bulletins not received by RTH Washington since 2011

NCO SS extremely responsive in restoring BUFR bulletins onto the GTS once the problem has been identified

As countries transition from TAC to BUFR, transition plans are not well documented nor well distributed. OPCs need:

Transition dates with overlapping TAC and BUFR for a test period

Names of corresponding BUFR bulletin headers to add to GTS subscription

There is significant concern that OPCs will lose observational data for DA as countries cease transmission of TAC bulletins and fully transition to BUFR unless:

BUFR Transition plans better communicated to OPCs

Observational data exchange between RTHs improved

Items of Concern

Other issues related to BUFR migration

ICAO has changed the BUFR migration plan METAR, SPECI, TAF, and SIGMET products will not

migrate to BUFR, but rather to XML format ICAO allows exchange of XML products starting Nov

2013 through bilateral agreements Mandated exchange in XML format by Nov 2019 Traditional Alphanumeric Code (TAC) products must

continue to be transmitted until 2019 The oceanographic community continues to resist the

transition to BUFR (as required by WMO for GTS)

NetCDF is the long established format for ocean data NDBC is converting multiple user’s Profiling Glider data

to BUFR and inserting these data on GTS for the ocean community

19

Significant challenges finding BUFR Bulletin listings (especially for US)

Listings for US BUFR bulletins are virtually non-existent through the WMO website

WIS/DAR at RTH Washington is confusing and not intuitive for the user for data discovery (i.e., Comprehensive list of Bulletins)

There have been random bulletin “drop-outs” where BUFR bulletins mysteriously stop being transmitted on GTS (Drifting Buoy (Global) & Synoptic Land observations (US)) Outages range from 5-7 days to several weeks

20

21

.

Status at FNMOC Continuing to add/modify BUFR decoder definition files as

required Adapting DA System to exploit additional (resolution,

metadata) info Identifying data paths (GTS/NOAAPort) for bulletins to

ensure receipt of bulletins is via primary data circuit (GTS) Partnering with NRL and WG/OD members to:

Identify metadata discrepancies between BUFR (internal) and existing databases

Identify BUFR encoding errors and reporting errors to the international community

22

.

Status at NCEP Negligible disruption for NCEP Central Operations NCEP EMC re-directing effort (expected several

months) to “Homogenize” BUFR data from different

international sources; Adapt to higher resolution data available in BUFR

than in TAC; Perform duplicate checking ACROSS (rather than

within) tanks

Status at NAVO Comparing ingested BUFR data vs what is ingested

at FNMOC Continuing to enhance BUFR ingest capability for

Ocean data Searching web for Oceanographic data available real-

time (Not on GTS). Continuing to encode releasable data in WMO

compliant formats for insertion onto the GTS

23

.

Status at 557th Weather Wing Initially anticipated negligible disruption for 557th WW

NWP production Already processing both BUFR and TAC data Crude duplicate elimination has been in place

(assimilate the first observation, reject any that follow for the same place and time)

As some countries/organizations transitioned, we found issues with our BUFR decoder. Developing improved decoder now.

More concern for downstream systems who have historically depended on TAC data disseminated from 557th WW

Assessing potential impact to Joint Environmental Toolkit (JET) at Operational Weather Squadrons

24

.

Upper Air BUFR Data Status

Migrating U.S. radiosonde data to BUFR Currently, the TAC TEMP data is converted to BUFR

Resulting output is low-resolution and contains numerous conversion and metadata errors, making it of limited benefit to users

High-resolution radiosonde data is available within the RRS (Radiosonde Replacement System) software at each NWS reporting site

Data is currently only sent to NCEI for archive on delayed schedule

OPCs have requested access to this data in real-time Expected to be of great benefit for use in NWP

A project is underway within NWS to obtain access to this data in real-time and convert it to a WMO-approved BUFR sequence for dissemination to worldwide OPCs via GTS

Project currently being managed by NCEP Central Operations

Converted data will meet WMO as well as OPC requirements

Project won’t include data from U.S. military (block 74) radiosonde sites, since those sites don’t have RRS installed

25

The Promise of BUFR

Non-operational high-resolution sounding for 72305 (Newport, NC) at 2015010800

BUFR: 1-sec (~5 m) data with balloon drift, all variables at all levels• 6373 levels from 1020 mb to 9.8 mb• Similar to operational soundings from EuropeTAC (TEMP): mix of mandatory and significant mass/wind levels• 16 mandatory, 120 significant mass, 36 significant wind levels

Launch500 mb

100 mb

9.8 mb

26

The Reality of BUFR

Structure of BUFR radiosonde data received at FNMOC• “Proper” BUFR—one message containing the full sounding (purple dots)• “BUFR by parts”—one message for each of the four TEMP parts (red squares)• NCO’s “legacy” BUFR uses BUFR by parts, as do New Zealand, Australia, China, Japan• NCO’s BMT BUFR provides multiple messages, some of which are proper BUFR

ECMWF, UKMO, NCEP do not plan to use BUFR by parts (red squares)

27

• Errors arising from decoding TAC prior to encoding in BUFR• Wrong leading digits for geopotential height• Negative 1000 mb heights incorrectly decoded• Various errors for tropopause level (variables mixed up, pressures wrong, etc.)• Incorrectly decoded pressure for significant levels with pressure > 1000 mb

• Other errors/problems in BUFR from TAC• Negative dewpoint depressions from India• Significant level winds from Vietnam divided by two• Inconsistencies in reported time/launch time in BUFR by parts• TAC errors propagated into BUFR (e.g., pressure divided by 10 for significant

winds above 100 mb from India)• Missing information

• Supplemental information for dropsondes not encoded in BUFR• Significant level winds from U.S. data in separate BUFR files—converted from

PPBB, PPDD PILOT format messages• Incomplete BUFR soundings with data filled in from TAC

• Truncation of significant level data from Australia• Missing data above 100 mb (e.g., South Africa)

Examples of Data Errors

28

Examples of Metadata Errors

• Failure to convert degrees-minutes-seconds to decimal degrees– e.g., 44373 (Dalanzadgad, Mongolia)

• WMO location (in degrees and minutes): 43 35N, 104 25E• BUFR location (in decimal degrees): 43.35°, 104.25° • FNMOC location (in decimal degrees): 43.58°, 104.42°

– All stations in Mongolia and the Philippines have this error

• Sign error in longitude– e.g., 70414 (Shemya AFB, AK): FNMOC = 174.10° ; BUFR = -174.11°– Also 60018 (Tenerife, Canary Islands), 71603 (Yarmouth, Nova Scotia)

• Confusion between Hp and Hha elevations– Hp = barometer height, which should be the elevation corresponding to the surface

pressure and so should be the one used for radiosonde data– Hha = either Ha (official airport altitude) or H (ground elevation)– e.g., 70350 (Kodiak, AK)

• WMO VolA gives Hp = 33.8 m and Hha = 5.5 m• BUFR uses 34 m = Hp, FNMOC uses 6 m = Hha• Hydrostatic estimates of surface elevation from the lowest mandatory level height favor FNMOC• Possible adjustment made to station pressure to correct the value to Hha?

29

Yuma Proving Ground (74004) is an example of a station with bad metadata in NCO’s station list and in the WMO VolA station list. YPG: 32.87, -114.33, 131 m FNMOC: 32.867. -114.333, 136 m BUFR: 32.5, -114.0, 231 m WMO: 32.5, -114.0, 231 m

The BUFR/WMO location is actually for 74005, with the latitude of 32° 50’ misinterpreted as 32.50 °.

Correct location

WMO, BUFR location

Metadata Error Example

30

.

Continuing BUFR Migration Monitoring and Collaboration

FNMOC, NAVO, NCEP, 557th WW -- Subgroup -- weekly conference calls – very active since the January

Sharing and comparing data and metadata error discoveries and tracking to final fix

Consulting data program managers (e.g. upper air) to pinpoint problems and find solutions

Federal Coordinator actively communicating interagency user needs and requirements to appropriate provider agencies

Finding appropriate U.S. and WMO data POCs to fix data issues – both U.S. and Global

31

.

Federated Gateway

WG/OD will assist by gathering representative proxy data to test operational effectiveness of the federated gateway

Product Distribution and Access (PDA)

WG/OD will assist in the transition and testing of products distributed by the DAPE Gateway to the PDA

WG/OD NESDIS member represents WG/OD at ongoing PDA meetings

Upcoming Activities

32

Thank you -- questions?