60
www.hdfgroup.org The HDF Group HDF/HDF-EOS Workshop XV 1 HDF-OPeNDAP Project Update Joe Lee and Kent Yang The HDF Group April 18, 2012
www.hdfgroup.org
The HDF Group
HDF/HDF-EOS Workshop XV 1
HDF-OPeNDAP Project UpdateJoe Lee and Kent Yang
The HDF Group
April 18, 2012
www.hdfgroup.org
Outline
Introduction to OPeNDAP
HDF5 Handler UpdateHDF4 Handler Update
Live Demo
HDF/HDF-EOS Workshop XV 2April 18, 2012
www.hdfgroup.org
Introduction to OPeNDAP
Goal: Background on Update
• Why OPeNDAP?
• What is HDF Handler?
• How important CF conventions?
April 18, 2012 HDF/HDF-EOS Workshop XV 3
www.hdfgroup.org
Access HDF via FTP
HDF/HDF-EOS Workshop XV 4
Users
FTP Clients
IDL
FTP Servers
HDF Files
AIRS/OMI
FTP
Local HDF Files
Aqua/Aura
April 18, 2012
www.hdfgroup.org
Client Library(libnc_dap)
Data Access Protocol (DAP) via Internet
Server(Hyrax)
Handler(hdf4/5_handler)
Remote Data(HDF4/5)
View Data (User)
OPeNDAP concept
5
Visualization Tools
(gradsdap)
Typical remote access problem
FTP/HTTP
Visualization tools
(e.g., GrADS)
HDF/HDF-EOS Workshop XVApril 18, 2012
www.hdfgroup.org
Access remote data easily
April 18, 2012 HDF/HDF-EOS Workshop XV 6
www.hdfgroup.org
without worrying about data plans.
April 18, 2012 HDF/HDF-EOS Workshop XV 7
www.hdfgroup.org
Visualize remote data easily, too.
April 18, 2012 HDF/HDF-EOS Workshop XV 8
www.hdfgroup.org
HDF handler is key
HDF/HDF-EOS Workshop XV 9
Users
OPeNDAP Clients
IDV
OPeNDAP Servers
HDF Files
Hyrax AIRS/OMI
DAP
NetCDF Java
Visualization Tools
HDF HANDLERAqua/Aura
April 18, 2012
www.hdfgroup.org
because if handler doesn’t follow CF
April 18, 2012 HDF/HDF-EOS Workshop XV 10
No Visualization Bonus!
www.hdfgroup.org
If handler gives up extra bonus
April 18, 2012 HDF/HDF-EOS Workshop XV 11
HDF / HDF-EOS
DAP Server
DAP
handler development is easier.
www.hdfgroup.org
What’s involved in extra bonus
April 18, 2012 HDF/HDF-EOS Workshop XV 12
DAP Server
CF Conventions
DAP
DAP Client
Visualization
Earth Scientist
Chec
k eve
ry st
ep.
HDF / HDF-EOS
www.hdfgroup.org
How handler developers feel
April 18, 2012 HDF/HDF-EOS Workshop XV 13
www.hdfgroup.org
Because we care all NASA products.
April 18, 2012 HDF/HDF-EOS Workshop XV 14
?
AURA
NPP
AQUATERRA
MISR
www.hdfgroup.org
Developers’ Dream
April 18, 2012 HDF/HDF-EOS Workshop XV 15
… all data are created equal,that they shall be visualized by all client tools easily…
www.hdfgroup.org
What if all people agreed…
April 18, 2012 HDF/HDF-EOS Workshop XV 16
UDF (Imaginary)
DAP Server
DAP
DAP Client
Visualization
Earth Scientist
development might have been much easier.
Check once.
Chec
k eve
ry st
ep
www.hdfgroup.org
Lessons from handler development
April 18, 2012 HDF/HDF-EOS Workshop XV 17
Communicate and work together among
different groups.
Standardize everything from data products
to visualization tools.
Follow conventions and standards.
www.hdfgroup.org
Outline
Introduction to OPeNDAP
HDF5-OPeNDAP Handler Update
HDF4-OPeNDAP Handler Update
Live Demo
HDF/HDF-EOS Workshop XV 18April 18, 2012
www.hdfgroup.org
Redesign of HDF5 handler
Goal: Support all NASA HDF5 products.
Modularize for future product support.
Follow CF conventions robustly.
April 18, 2012 HDF/HDF-EOS Workshop XV 19
Product
www.hdfgroup.org
Why did we redesign?
April 18, 2012 HDF/HDF-EOS Workshop XV 20
In 2007,
was enough forAURAGrid
HDF5 handler
www.hdfgroup.org
Why did we redesign?
April 18, 2012 HDF/HDF-EOS Workshop XV 21
In 2011, wasn’t enough for
AURAGrid
AURASwath
AURAZA GSSTF
www.hdfgroup.org
Why did we redesign?
April 18, 2012 HDF/HDF-EOS Workshop XV 22
In 2011, wasn’t enough for
AURAGrid
AURASwath
AURAZA GSSTF
www.hdfgroup.org
and more… so we gave up!
April 18, 2012 HDF/HDF-EOS Workshop XV 23
GOSAT
ICESat-2
Aquarius
SMAP
www.hdfgroup.org
We rebuilt one that can scale up
April 18, 2012 HDF/HDF-EOS Workshop XV 24
www.hdfgroup.org
by modularizing the design
April 18, 2012 HDF/HDF-EOS Workshop XV 25
HDF5 handler
CF
HDF5
GOSAT Aquarius SMAP
HDF-EOS5
GRID SWATH ZA
Generic
with a big room for future products.
www.hdfgroup.org
Key updates on HDF5 handler
April 18, 2012 HDF/HDF-EOS Workshop XV 26
version 1.5.x 1.6.x
CF-conventionSupport
Primitive Robust
Range of NASA Data Products
Limited All
Mulitiple Grids/Swath/ZA
No Yes
ConfigurationOptions
Static Build Run-time
Support forFuture Products
Rigid Flexible
www.hdfgroup.org
HDF-EOS5 products supported
• OMI Swath, Grid• MLS Swath• HIRDLS Swath, ZA• TES Swath, Grid• GSSTF
April 18, 2012 HDF/HDF-EOS Workshop XV 27
www.hdfgroup.org
HDF5 products supported
• Aquarius Grid• MEaSUREs SeaWiFS DeepBlue• SBUV2 Ozone (O3) Profile• SMAP (Simulation)
April 18, 2012 HDF/HDF-EOS Workshop XV 28
www.hdfgroup.org
HIRDLS Zonal Average
April 18, 2012 HDF/HDF-EOS Workshop XV 29
www.hdfgroup.org
New TES Swath
April 18, 2012 HDF/HDF-EOS Workshop XV 30
www.hdfgroup.org
MEaSUREs GSSTF
April 18, 2012 HDF/HDF-EOS Workshop XV 31
www.hdfgroup.org
Aquarius Grid (Simulation)
April 18, 2012 HDF/HDF-EOS Workshop XV 32
www.hdfgroup.org
MEaSUREs SeaWiFS DeepBlue
April 18, 2012 HDF/HDF-EOS Workshop XV 33
www.hdfgroup.org
Ozone Profile (BUV)
April 18, 2012 HDF/HDF-EOS Workshop XV 34
www.hdfgroup.org
SMAP Swath (Simulation)
April 18, 2012 HDF/HDF-EOS Workshop XV 35
www.hdfgroup.org
SMAP Swath (Simulation)
April 18, 2012 HDF/HDF-EOS Workshop XV 36
www.hdfgroup.org
No CF magic for some products
Tool / CF Limitation
• GOSAT/acos: 1-D trajectory
• ICESat-2
• OMI L2G: Lat/Lon with 3-D
Java-NetCDF Client Limitation
• Dataset w/ long string
April 18, 2012 HDF/HDF-EOS Workshop XV 37
www.hdfgroup.org
It doesn’t mean handler failure
April 18, 2012 HDF/HDF-EOS Workshop XV 38
becauseother tools like
can still access and visualize data.
www.hdfgroup.org
MATLAB and IDL will work too.
April 18, 2012 HDF/HDF-EOS Workshop XV 39
HDF5 Examples under zoo will work.
• Visit http://hdfeos.org/zoo
• Replace local file access with OPeNDAP
URL access.
MATLAB: Try new OPeNDAP functions.
• The rest will be same.
www.hdfgroup.org
Read if you can’t see
April 18, 2012 HDF/HDF-EOS Workshop XV 40
http://acdisc.sci.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMTO3e.003/2012/OMI-Aura_L3-OMTO3e_2012m0416_v003-2012m0418t020608.he5.ascii?lon[1029:1:1029],lat[513:1:513],_HDFEOS_GRIDS_OMI_Column_Amount_O3_Data_Fields_ColumnAmountO3[513:1:513][1029:1:1029]
Dataset: OMI-Aura_L3-OMTO3e_2012m0416_v003-2012m0418t020608.he5
lon, 77.375 lat, 38.375 _HDFEOS_GRIDS_OMI_Column_Amount_O3_Data_Fields_ColumnAmountO3.lon, 77.375 _HDFEOS_GRIDS_OMI_Column_Amount_O3_Data_Fields_ColumnAmountO3._HDFEOS_GRIDS_OMI_Column_Amount_O3_Data_Fields_ColumnAmountO3[_HDFEOS_GRIDS_OMI_Column
_Amount_O3_Data_Fields_ColumnAmountO3.lat=38.375], 309.6
www.hdfgroup.org
or hear from Siri?
April 18, 2012 HDF/HDF-EOS Workshop XV 41
www.hdfgroup.org
Outline
• Introduction
• HDF5-OPeNDAP Handler Update
• HDF4-OPeNDAP Handler Update
• Live Demo
HDF/HDF-EOS Workshop XV 42April 18, 2012
www.hdfgroup.org
HDF4 OPeNDAP handler update
• Corrected the handling the scale/offset for MODIS products.
• Added the support for Lambert azimuthal equal-area (LAMAZ).
• Added the support for Space-oblique Mercator projection (SOM).
• Other minor bug fixes.
HDF/HDF-EOS Workshop XV 43April 18, 2012
www.hdfgroup.org
Key updates on HDF4 handler
April 18, 2012 HDF/HDF-EOS Workshop XV 44
version 3.9.3 3.9.4
Some MODISScale/offset
Wrong Corrected
LAMAZ projection
support
No Yes
SOM projection
support
No Yes
www.hdfgroup.org
MODIS scale / offset correction
HDF/HDF-EOS Workshop XV 45April 18, 2012
www.hdfgroup.org
Support for LAMAZ projection
HDF/HDF-EOS Workshop XV 46April 18, 2012
www.hdfgroup.org
Support for SOM projection
HDF/HDF-EOS Workshop XV 47April 18, 2012
Dataset { UInt16 Blue_Radiance_RDQIU1[SOMBlockDimD1 = 180][XDimD2 = 128][YDimD3 = 512]; Int32 SOMBlockDimU2[SOMBlockDimD1 = 180]; Float32 LatitudeU3[XDimD2 = 128][YDimD3 = 512]; Float32 LongitudeU4[XDimD2 = 128][YDimD3 = 512];…}
Dataset { UInt16 Blue_Radiance_RDQIU1[SOMBlockDimD1 = 180][XDimD2 = 128][YDimD3 = 512]; Int32 SOMBlockDimU2[SOMBlockDimD1 = 180]; Float32 LatitudeU3[SOMBlockDim = 180][XDimD2 = 128][YDimD3 = 512]; Float32 LongitudeU4[SOMBlockDim = 180][XDimD2 = 128][YDimD3 = 512];…}
www.hdfgroup.org
HDF-EOS2 products supported
GES DISC• AIRS / MERRA / TOMS
LAADS / LP DAAC / NSIDC• Many MODIS products
NSIDC• AMSR_E / NISE / MODIS
LaRC• MISR / MOPITT / some CERES (TRMM)
HDF/HDF-EOS Workshop XV 48April 18, 2012
www.hdfgroup.org
HDF4 products supported
GES DISC• TRMM L1B, L2B Swath • TRMM L3 Grid 42B & 43B products
OBPG (Ocean Color)• SeaWiFS / MODIST / MODISA• CZCS / OCTS
HDF/HDF-EOS Workshop XV 49April 18, 2012
www.hdfgroup.org
HDF4 products supported
CERES• CER_AVG• CER_ES4• CER_ISCCP-D2like-Day• CER_ISCCP-D2like-GEO• CER_SRBAVG• CER_SYN• CER_ZAVG
HDF/HDF-EOS Workshop XV 50April 18, 2012
www.hdfgroup.org
HDF4 handler limitations
• HDF-EOS2 + additional HDF4 objects• In some products, some additional HDF4
objects will be ignored.
• Other HDF4 products not listed in previous slides are not tested and may not be visualized by Java OPeNDAP clients.
HDF/HDF-EOS Workshop XV 51April 18, 2012
www.hdfgroup.org
Other limitations
CF conventions Limitation
• _FillValue inside latitude and longitude
Java-NetCDF Limitation
• Java Heap Memory
HDF/HDF-EOS Workshop XV 52April 18, 2012
www.hdfgroup.org
Outline
• Introduction
• HDF5-OPeNDAP Handler
• HDF4-OPeNDAP Handler
• Live Demo
HDF/HDF-EOS Workshop XV 53April 18, 2012
www.hdfgroup.org
We’ve updated handlers
HDF/HDF-EOS Workshop XV 54April 18, 2012
OPeNDAPHDF
Handlers
HDF-EOS
2
HDF5
HDF-EOS
5
HDF4
… and will continue.
www.hdfgroup.org
GES DISC Uses Handlers
HDF/HDF-EOS Workshop XV 55April 18, 2012
www.hdfgroup.org
Please try HDF4/HDF5 handlers
HDF/HDF-EOS Workshop XV 56April 18, 2012
New hdf5_handler will be released in July.
Source codes and RPMs:
http://opendap.org
Demo HDF5/HDF4 servers:
http://hdfeos.org/zoo/hdf4_handler/
http://hdfeos.org/zoo/hdf5_handler/
www.hdfgroup.org
and give us feedback.
HDF/HDF-EOS Workshop XV 57April 18, 2012
www.hdfgroup.org
The HDF Group
HDF/HDF-EOS Workshop XV 58
Thank you!
April 18, 2012
www.hdfgroup.org
Acknowledgements
HDF/HDF-EOS Workshop XV 59April 18, 2012
This work was supported by Subcontract number 114820 under Raytheon Contract number NNG10HP02C, funded by the National Aeronautics and Space Administration (NASA) and by cooperative agreement number NNX08AO77A from the NASA. Any opinions, findings, conclusions, orrecommendations expressed in this material are those of the authors and do not necessarily reflect the views of Raytheon or the National Aeronautics and Space Administration.
www.hdfgroup.org
The HDF Group
Questions/comments?
HDF/HDF-EOS Workshop XV 60April 18, 2012
