42
HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)
HDF5 OPeNDAP Project
MuQun Yang and Hyo-Kyung Lee (THG)
James Gallagher (OPeNDAP, Inc.)
Question of the Day
• How can I get One Sub from SUBWAY?• Chicken teriyaki with no onion
• How can I get One Subset of EOS data image from NASA?• Today’s satellite image right above the roof
Subway
• Easy: anywhere / accepts Visa|MC
• Efficient: your own recipe• Cool: eat fresh and look good
How can I get One Sub from SUBWAY?
Dumb-way
• Difficult: bring a USB drive and visit NASA
• Inefficient: download 10 Gbyte and search for 10 byte dataset
• Not cool: wrong or useless dataset
How can I get One Subset of EOS data image from NASA?
DAP-way!
• Easy: anywhere / accepts IE|FF
• Efficient: WYSIWYG – less fat!
• Cool: Visualization clients – see fresh!
How can I get One Subset of EOS data image from NASA?
DAP-way Implementation
• OPeNDAP: Free DAP-way
Software
• Server-Client Model APIs
• Widely used by Earth Science
Community
OPeNDAP
• Market Dominance:
•
• http://www.opendap.org
OPeNDAP server sites
Courtesy of OPeNDAP, Inc.
OPeNDAP Servers
Data Data Data Data Data Data Data
Matlab
HDF4 JDBC
FreeFromFITS
CDF CEDAR
Data
netCDF
netCDF HDF4 Matlab
Data
DSP
DSP
Data
JGOFS
Tables SQL FITS CDFFlat
Binary CEDAR
Data
CODAR
Data
ESML
GeneralCODAR
Courtesy of OPeNDAP, Inc.
OPeNDAP Clients
netCDF C netCDF Java
IDVFerret GrADS VisAD ncBrowse Matlab ExcelIDL Access
MatlabClient
IDLClient
Courtesy of OPeNDAP, Inc.
How OPeNDAP works
OPeNDAP Client
OPeNDAPHDF4 Server
HDF4
OPeNDAPNetCDF Server
NetCDF
Architecture of OPeNDAP Server• Server 3 (CGI)
• C/C++ based HTTP Server• Insecure / Inflexible
• Server 4 (Hyrax)• Java based TomCat Servlet Engine
• Secure / Flexible
Server 3 (CGI) Architecture
OLFS
Java Servlet Engine
BES
Unix Daemon
BES Commands
XML- encapsulated object
DataStore
File system with data files like HDF, NetCDF and SQL Database, …
DAP
XML, GIS, KML
HTML
Optional catalogs: XML, GIS, KML
Server 4 (Hyrax) ArchitectureHTTP Server
Courtesy of OPeNDAP, Inc.
http://docs.opendap.org/index.php/Hyrax
Example Usage
OPeNDAP Client
OPeNDAPHDF4 Server
HDF4
Syntactic Structure of DataSemantic Meaning of Data
Actual Content of Data
OPeNDAP Visualization Client
What about HDF5?
OPeNDAP Client
OPeNDAPHDF4 Server
HDF4
OPeNDAPNetCDF Server
NetCDF
OPeNDAPHDF5 Server
HDF5
Why Important?
• HDF5: NASA EOS / NPOESS
• NASA’s MISSION: ACCESS!
• Our MISSION: Build DAP-way Bridge
Goals
•1st: Transform HDF5 access in DAP-way
• 2nd: Yet preserve the beauty of HDF5
OPeNDAP HDF5 Server
1. Map HDF5 Datatypes in DAP-way
2. Plus, tweaks for OPeNDAP Clients
3. Plus, tweaks for HDF-EOS Files
Goal: Transform HDF5 access in DAP-way
Mapping HDF5 in DAP-way• Prototype server in 2001
• NASA Grant from Nov. 2006
• Support for Compound Datatype
• Support for Group
• Support for References / Links
• First Product Release in Mar. 2008
OPeNDAP HDF5 Server
1. Map HDF5 Datatypes in DAP-way
2. Plus, tweaks for OPeNDAP Clients
3. Plus, tweaks for HDF-EOS Files
Goal: Transform HDF5 access in DAP-way
Subway Customers
• I want 3 inch-long sub
• I need Wasabi sauce
• I eat only wheat bread
• I eat only meat balls
OPeNDAP Clients
• I hate foot-long variable names.• I need special attributes on dataset.• I care pre-defined Grid data type.• I care only well-formed attributes.
Not all OPeNDAP clients are created equal!
Tweaks for OPeNDAP Clients Two configuration options• --enable-short-name = cut bread• --enable-CF = put Wasabi sauce
OPeNDAP HDF5 Server
1. Map HDF5 Datatypes in DAP-way
2. Plus, tweaks for OPeNDAP Clients
3. Plus, tweaks for HDF-EOS5 Files
Goal: Transform HDF5 access in DAP-way
Subway Customers
• I want 3 inch-long
• I need Wasabi sauce
• I eat only wheat bread
• I eat only meat balls
Subway Suppliers Problem
• No breads, only wheat!
• No meat balls, only meat!
• I eat only wheat bread
• I eat only meat balls
Subway Customers - OK
HDF EOS
• Grid with No Dimension data • Clients expect Grid with Dimension data
• Metadata Attribute in Two Strings• Clients expect One Structured Format
Not all HDF5 files are created for DAP-way!
Tweaks for HDF-EOS Two more configuration options• --enable-eos-grid = bake bread• --enable-eos-meta = make balls
OPeNDAP HDF5 Server
1. Map HDF5 Datatypes in DAP-way
2. Plus, tweaks for OPeNDAP Clients
3. Plus, tweaks for HDF-EOS5 Files
Goal: Transform HDF5 access in DAP-way
Day After Server Tweaks• Finally, Happy Clients!
GrADSNCLFerretMATLAB
ncBrowse
IDVODC
Live DemoOzone concentration level over the south pole
Goals
•1st: Transform HDF5 access in DAP-way
• 2nd: Yet Preserve the beauty of HDF5
HDF5 Served in DAP-way (Yet Preserve the beauty of HDF5)
• There are some things money can’t buy. For everything else, there is MasterCardTM.
• There are some things DAP can’t serve. For everything else, there is _________?
Minimize
Maximize
Some Things
• Hard: Opaque, Bitmap, Enum, 64 bit Integer, Variable Length types
• Illegal: Reserved characters in DAP are used in Dataset/Group name in HDF5
(that OPeNDAP HDF5 server can’t serve)
For Everything Else
• Let there be HDF5-friendly OPeNDAP client library!
• Package and Deliver in DAP-way first.• Then, let the client library handle it.• Concept of Trojan Horse
Example: Group in HDF5Traditional OPeNDAP client library: It’s an attribute that I don’t understand. I’ll ignore it.
HDF5-friendly OPeNDAP client library: I was waiting for this key attribute to re-construct HDF5
Example: Reference in HDF5
• Object / Regional Reference • Map to DAP URL at server• No de-referencing of URL at client library• Important for NPOESS
HDF5-Friendly OPeNDAP Client Library
• One more reason: Tame Clients • Only ODC can display Swath properly.
• Easy but Evil OPeNDAP NC Client Library! easy: nc style API evil: fixed dimensional attributes
• Our client library must be Easy but Good
No Latitude and Longitude
Courtesy of NASA
libnc-dap
???
liboc-dap
NC-Friendly OPeNDAP Client Library
libdap
dapserver
OPeNDAP NC Server
GrADS
NetCDFView NetCDFHDF5
Group/Ref/Swath
OPeNDAP HDF5 Server
View HDF5
GrADS
View G/R/Swath
HDF5-Friendly OPeNDAP Client Library
Summary
• DAP-way access of HDF5 Is Easy / Efficient / Cool Loses some things Requires HDF5-friendly OPeNDAP Client Library
• HDF5-friendly OPeNDAP Client Library Serves HDF5 better Tames evil visualization clients
Future Work• HDF5 DAP Mapping Document
• Finish HDF5-friendly OPeNDAP Client Library prototype
• Test it on GrADS and display Swath
Project Website http://hdfdap.hdfgroup.uiuc.edu/joomla
• Beta preview• Feedbacks are more than welcome!
Credits
Mike Folk (THG)Robert McGrath (NCSA)
Peter Leonard, Daniel Kahn, Marghi Hopkins (ADNET)Christopher Lynnes, James Johnson, Denis Nadeau (NASA)
Jennifer Adams (GrADS)Dave Brown (UCAR)
