HDF OPeNDAP Project Update

HDF OPeNDAP Project Update

MuQun Yang and Hyo-Kyung Lee

The HDF Group

March 31, 2009

1Annual briefing to ESDIS04/19/23

Outline

• HDF5 – OPeNDAP work review• HDF4 – OPeNDAP handler progress report• Future work plan

04/19/23 Annual briefing to ESDIS 2

Users

OPeNDAP Clients

IDV

OPeNDAP Servers

HDF4 Files

Hyrax AIRS

DAP

libnc-dap

Visualization Tools

Access data via OPeNDAP

HDF HANDLER

04/19/23 3Annual briefing to ESDIS

HDF5 – OPeNDAP Work


HDF5-OPeNDAP handler

• Update with new Hyrax• Help GES DISC serve TES and OMI data• Finish HDF5 – DAP2 mapping document and

submit it as a technical note to NASA SPG



HDF5-Friendly OPeNDAP Client Library

• Retrieve the key HDF5 information• Easy to access other HDF5 data (e.g. EOS5

swath)

Example: Groups in HDF5Traditional OPeNDAP client library: It’s an attribute that I don’t understand. I’ll ignore it.

HDF5-Friendly OPeNDAP client library: I was waiting for this key attribute to re-construct HDF5



MLS swath via GrADS

CO at these elevations is a measure of pollution

lower stratosphere

upper stratosphere

Before

After

CO level near Beijing, China before, during and after Olympics

HDF4 – OPeNDAP Work


Current HDF4-OPeNDAP handler

• Contributed by JPL in 1998• Part of the OPeNDAP distribution• Visualization client tools cannot display most

HDF-EOS2 and HDF4 data


• Enhance the HDF4-OPeNDAP handler

• More NASA HDF-EOS2 and HDF4 data can be accessed via visualization clients


Our work

Challenges – more products


Hundreds of HDF-EOS2 and HDF4 Data products

Challenges – Requirement from Clients

• CF conventions

- Clients expect a certain data structure

- Clients expect some key attributes


• Various HDF-EOS2 Grid projections• Geographic• Sinusoidal• Polar stereographic• …


Challenges – Grid

Data [4][12]Lon[12]Data [4][8]Lon[4][8]

Geographic

Sinusoidal

• HDF-EOS2 Swath- Dimension map


Challenges - SwathData field has three columnsLongitude field has two columns Longitude should have three columns

Challenges – various products • “Special” HDF-EOS2 files

• Hybrid HDF-EOS2 files- Some HDF4 objects that cannot be read by EOS

library

• HDF4 data products


Enhanced HDF4 Handler


• Follow CF conventions Generate the correct DAP data type. Add shared geo-location information. Insert additional key attributes.

How: Parse Metadata or Use EOS2 library

Implementation choice

• Use Metadata parser or HDF-EOS2 library• Metadata parser

• Pro: simple

• Con: limited products

• HDF-EOS2 library• Pro: maximum products

• Con: install HDF-EOS2 library first


Solution Comparison


Quick Implementation

• Parse Metadata – hdf5_handler

• Use HDF-EOS2 library – h4toh5


Current Status

• MODIS / AIRS Grid and Swath

• Other Projections (Polar, Sinusoidal)

• Testing


HDF-EOS2 in CF via DAP

2204/19/23

AIRS Grid AIRS.2008.10.27.L3.RetStd001.v5.2.2.0.G08303124144.hdfhttp://daac.gsfc.nasa.gov/daac-bin/whom/mk_page_cgi.pl?PATH=datapool/AIRS/Level3_V005/01_AIRX3STD/2008/10MODIS GridMOD08_D3.A2008284.005.2008289105354.hdfhttp://modis-atmos.gsfc.nasa.gov/MOD08_D3/acquiring.html

AIRS Swath ftp://airspar1u.ecs.nasa.gov/data/s4pa//Aqua_AIRS_Level2/AIRX2RET.005//2008/330/

AIRS.2008.11.25.143.L2.RetStd.v5.2.2.0.G08331121849.hdfMODIS Swath

MOD06_L2.A2001222.0025.004.2003127111841.hdfhttp://modis-atmos.gsfc.nasa.gov/MOD06_L2/acquiring.htmlAnnual briefing to ESDIS

More challenges: Missing Attributes

• MODIS Grid – Rain fall


No Unit!

• AURA OMI Grid - Ozone

More challenges Match dimension names

• Horizontal geolocation information:

- Non-standard names ( e.g. nlon, nlat)

How to match?• Vertical geolocation information:

- Users provide the geolocation name and vaule but no conventions are enforced.

How to know?


More challenges – Missing Values• In a swath file, a data field has three dimensions. But no dimension value for the third dimension.


MISR_AM1_TC_CLASSIFIERS_P020_O000995_F06_0011.hdf

A Solution to challenges

• Provide an option for the handler to generate fully CF-compliant output

• Create a separate XML input file for the handler and feed it into the handler

• Insert attributes, renaming rules, and missing values by putting them in the XML input

• This requires a knowledge about the characteristics of data in advance


One Solution An input file from data provider

• Example: Attributes, Renaming, Values


<xml><variable name=‘RrLandRain’> <attr name=‘unit’ type=‘String’> inches</attr></variable><rename from=‘pressStd’ to=‘level’></rename><insert name=‘NAltitudeDim’ size=10 type=‘Int8’>0,1,2,3,4,5,6,7,8,9</insert></xml>

Future Work (Short Term)

• Expand test-suite with data pool

• Release through OPeNDAP website

• Support for Swath via Parsing

• Support for Point data type via HDF-EOS2 library


Future Work (Long Term)

• General HDF4 Files Support

• Using an input XML file to provide full

CF support

• HDF-EOS2 to DAP2 Mapping Document


Thank You!

October 15-18, 2008 Workshop Name (edit footer to change) 30

Acknowledgements

This work was supported by the Cooperative Agreement with the National Aeronautics and Space Administration (NASA) under NASA grant NNX06AC83A, NNX08A077A and NNX06AG75A.

Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NASA.

October 15-18, 2008 31Workshop Name (edit footer to change)

Questions/comments?

October 15-18, 2008 Workshop Name (edit footer to change) 32

Date post:	30-Dec-2015
Category:	Documents
Upload:	beatrice-wallace
View:	45 times
Download:	1 times

HDF OPeNDAP Project Update

Documents