HDF5 OPeNDAP ProjectUpdate and Demo

MuQun Yang and Hyo-Kyung Lee (The HDF Group)

James Gallagher (OPeNDAP, Inc.)

1HDF and HDF-EOS Workshop XII10/17/2008

OPeNDAP Review

• A software framework that allows simple access to remote data

• Data Access Protocol• Client-server model• Widely used by Earth Science Community

2HDF and HDF-EOS Workshop XII10/17/2008

Client Library(libnc-dap)

DAP Protocol (via http)

Server(apache)

Handler(hdf4_handler)

Remote Data(HDF4)

View Data (User)

OPeNDAP Concept

3HDF and HDF-EOS Workshop XII10/17/2008

Visualization Tools

(gradsdap)

Typical Remote Access

FTP/HTTP

Visualization tools

(e.g. grads)

DAP Protocol (via http)

Server(apache)

Remote HDF5View Data (User)

OPeNDAP HDF5 Handler

4HDF and HDF-EOS Workshop XII10/17/2008

HDF5 Handler

Mapping HDF5 Datatypes to DAP Datatypes

• Compound Datatype

• Groups

• Object/Regional References

5HDF and HDF-EOS Workshop XII10/17/2008

Challenges

DAP Protocol (via http)

Server(apache)

View Data (User)

OPeNDAP HDF5 Handler with HDF-EOS5

6HDF and HDF-EOS Workshop XII10/17/2008

HDF5 Handler

Remote HDF5

Client Library(libnc-dap)

Visualization tools

(grads)

Remote HDF-EOS5

Challenges: HDF- EOS5

• Grid with No Geolocation data Clients expect Grid with Geolocation data

• Some Attributes stored as Extremely Long String(s) (e.g. StructMetada.0) Clients expect structured attributes

7HDF and HDF-EOS Workshop XII10/17/2008

Tweaks for HDF-EOS5

Added two default HDF5 handler configuration options:• --enable-eos-grid • --enable-eos-meta

8HDF and HDF-EOS Workshop XII10/17/2008

Challenges: OPeNDAP Clients

• Need special attributes on dataset.• Need shared geolocation variables outside Grid.

9HDF and HDF-EOS Workshop XII10/17/2008

Tweaks for OPeNDAP Clients

Added two optional handler configuration options:• --enable-short-name• --enable-CF

10HDF and HDF-EOS Workshop XII10/17/2008

HDF5 Handler

DAP Protocol (via http)

Server(apache)

View Data (User)

OPeNDAP HDF5 Handler with HDF-EOS5

11HDF and HDF-EOS Workshop XII10/17/2008

HDF5 Handlerw/ tweaks

Remote HDF5

Client Library(libnc-dap)

Visualization tools

(grads)

Remote HDF-EOS5

Day After Server Tweaks

• Finally, Happy Clients!

GrADSNCLFerretMATLAB

ncBrowse

IDVODC

IDL

12HDF and HDF-EOS Workshop XII10/17/2008

Problems of Tweaking HDF5 Handler

13HDF and HDF-EOS Workshop XII10/17/2008

Remember that we added two optional HDF5 handler configuration options:1. --enable-short-name2. --enable-CF

Causes ambiguity among variable names

(e.g. /GroupA/ozone vs. /GroupB/ozone)

Drops some key attributes (e.g. StructMetdata, HDF_ROOT_GROUP)

HDF5-Friendly OPeNDAP Client Library

14HDF and HDF-EOS Workshop XII10/17/2008

DAP Protocol (via http)

Server(apache)

View Data (User)

HDF5 Handler

Client Library(libnc-dap)

Visualization tools

(grads)

Remote HDF5

HDF5 Handlerw/ tweaks

HDF5-FriendlyClient Library(liboc-dap)

HDF5 GroupsView Groups???

Example: Groups in HDF5Traditional OPeNDAP client library: It’s an attribute that I don’t understand. I’ll ignore it.

HDF5-Friendly OPeNDAP client library: I was waiting for this key attribute to re-construct HDF5

15HDF and HDF-EOS Workshop XII10/17/2008

Example: Reference in HDF5

• Important for NPOESS• Object / Regional Reference • Map to DAP URL

16HDF and HDF-EOS Workshop XII10/17/2008

Example: dap2h5

• A test application for the client library• It can construct HDF5 from DAP output

17HDF and HDF-EOS Workshop XII10/17/2008

DAP Protocol (via http)

Server(apache)

View Data (User)

HDF5-F. Client Library App.

(dap2h5)

Remote HDF5

HDF5 Handler

HDF5-FriendlyClient Library

(liboc-dap)

Group/Ref.View Group/Ref.

• One more reason: Help Clients to view Swath

HDF5-Friendly OPeNDAP Client Library

18HDF and HDF-EOS Workshop XII10/17/2008

HDF5 Handler

DAP Protocol (via http)

Server(apache)

View Data (User)

Visualizing HDF-EOS5 Grids

19HDF and HDF-EOS Workshop XII10/17/2008

HDF5 Handlerw/ tweaks

Remote HDF5

Client Library(libnc-dap)

Visualization tools

(grads)

Remote HDF-EOS5

Swath

Visualizing HDF-EOS5 Swath Problem

???

HDF5-FriendlyClient Library(liboc-dap)

Visualization Tools

(gradsoc)

View Swath

Demo: MLS swath via GrADS

• GrADS coupled with our client library

• The client library provides grid mapping from swath data

• GrADS displays swath through HDF-EOS5 specific client library API calls

20HDF and HDF-EOS Workshop XII10/17/2008

Summary of Client Library Prototype

• Finished coding • Tested with the GrADS visualization

client• Documented • Working on a demo DAP to HDF5

tool by using the client library prototype

• Will test with NCL if time allows

10/17/2008 HDF and HDF-EOS Workshop XII 21

Caution

• Our Client Library is ONLY PROTOTYPE! It does NOT support all DAP data types It does NOT support all AURA files It does NOT support all Visualization clients

22HDF and HDF-EOS Workshop XII10/17/2008

Summary

• HDF5 Access via OPeNDAP Is Easy Is used by GES DISC to serve Aura files

• HDF5-Friendly OPeNDAP Client Library Prototype Serves HDF5 better(EOS swath) Benefits visualization clients

23HDF and HDF-EOS Workshop XII10/17/2008

Future Work

• HDF5 to DAP2 Mapping Document• Release HDF5-friendly OPeNDAP Client Library Prototype

• URL: http://hdfgroup.org/projects/opendap

24HDF and HDF-EOS Workshop XII10/17/2008

Credits

Mike Folk (The HDF Group)Robert McGrath (NCSA)

Peter Leonard, Daniel Kahn, Marghi Hopkins (ADNET)Christopher Lynnes, James Johnson, Denis Nadeau, Fan Fang

(NASA)Jennifer Adams (GrADS)

Dave Brown (UCAR)

25HDF and HDF-EOS Workshop XII10/17/2008

Acknowledgement

This work was supported basing upon the Cooperative Agreement with the National Aeronautics and Space Administration (NASA) under NASA grant NNX06AC83A and NNX06AG75A. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NASA. 

26HDF and HDF-EOS Workshop XII10/17/2008