OPeNDAP Services for ESG September 25, 2014

1

Earth System Grid Center for Enabling Technologies

OPeNDAP Services for ESG

April 21, 2023

Peter Fox, Patrick West, Stephan ZednikRPI

Performance Measures x.x, x.x, and x.x

2

Earth System Grid Center for Enabling Technologies: (ESG-CET)

In ESG II (in regard to data)

Server side aggregation DAP object transfer via HTTP and GridFTP GSIFTP integration, myProxy support OPeNDAP-g -> led to architecture of OPeNDAP/ Hyrax

(primarily BES is from OPeNDAP-g)

3


Requirements leading to OPeNDAP-g

Separation of the core Data Access Protocol (DAP) from the transport protocol (HTTP).

High Performance Computing. The previous CGI based servers did not have the capacity required by ESG. Error and memory handling added.

Security. Once the OPeNDAP was independent of the transport protocol, adding security was possible by relying on the Globus gsiFTP system.

Aggregation. OPeNDAP 3.0 did not operate on aggregated datasets. OPeNDAP-g does.

Transport protocol independence and HPC were incorporated back into OPeNDAP leading to the current version. Security and aggregation remain an ESG only feature.

4


OPeNDAP-g Architecture

Dispatcher BES

OPeNDAP-g Dispatcher (e.g. ESG Front-end Server) Receives requests and asks the BES to fill them Uses Apache Modules Does not directly ‘touch’ data, handles URLs

Data

Back End Server (BES) Reads data files, Databases, et c., returns info May return DAP objects or other data (netCDF)

55

OPeNDAP Hyrax Architecture

OLFS BES

OPeNDAP Lightweight Front end Server (OLFS) Receives requests and asks the BES to fill them Uses Java Servlets Does not directly ‘touch’ data Multi-protocol

Data

Back End Server (BES) Reads data files, Databases, et c., returns info May return DAP2 objects or other data Does not require web server

Client

66

GridFTPDAP2

GridFTPDAP2

HTTPDAP2HTTPDAP2

ASCII outputASCII output

HTML formHTML form

Info outputInfo output

OPeNDAP Lightweight Front end ServerOPeNDAP Lightweight Front end Server

THREDDS**THREDDS**

Request Formulation**Request Formulation**

Req

ues

t fr

om

cli

ent

Res

po

nse

to

cli

ent

BE

S

SOAP-DAP (HTTP)

DAP2 (GridFTP, HTTP)

RDF, OWL, JSON (HTTP)

PML outputPML output

77

Hyrax/ Back-end Server

Network Protocol andProcess start/stopactivities

Data Store Interfacese.g. IOSP

BES Framework

PPT/PPTS*Initialization/Termination

DAP2Access

NetCDF3 HDF4 RDF/ SPARQL…

Provenance

Commands**e.g. server-side, Ferrett

BES Commands/ XML Documents

*PPT is built in (other protocols)**Some commands are built in

Data DataData

DataCatalogs

8


OPeNDAP-g services for ESG

Data access via Dataportals. In this Use Case users interact with the portal, browse the catalogs and decide what data to download. The portal passes the request to OPeNDAP-g which executes it and returns the data to the portal. The portal returns a URL to the user to download the data. Data access via netCDF library. In this Use Case users link their applications with OPeNDAP-g client library for netCDF. The user can “open” a URL that refers to ESG data. The library fetches the data for the user and makes it a local netCDF file The performance requirements for ESG II were met. However, ESG-CET scales up these requirements.

9


Status of the Community OPeNDAP Server Software

Together the OLFS and the BES are known as Hyrax Hyrax 1.6 provides support for NcML-based

aggregation Data response streamed back as netCDF file RDF response type Updated DDX response type (Data Definition XML) Beginning development of DataDDX – multi-mime

response with data and DDX Full security audit and static code analysis certification

to comply with NOAA and NASA requirements

10


ESG-CET and data

Large data sets, numbers and sizes• High performance• Flexible architecture, both client and several types and

numbers of servers• Aggregation• Server side operations• Multiple transport protocol options

Full ESG security support as well as loose federation Read-only client access via API (netCDF/CDM) To satisfy the new goals, the OPeNDAP services for ESG have

been re-architected. We now use parts of the standard OPeNDAP framework Hyrax,

focusing on high performance for the client side and extended flexibility.

11

ESG-CET and Products (server side functions)

Goal: drop in replacement for the TDS part of FTDS in LAS

Requires netCDF-Java Input-Output Service Provider (IOSP) adapter for Hyrax/BES

Use case examination will be required


12

Security Infrastructure status

OPeNDAP BES Security• SSL authentication between gridFTP middle tier and the

BES. No persistent SSL connections are maintained RNI Integration with ESG Security Infrastructure

• RNI client supports gsiFTP connections to ESG GridFTP servers

• ESG GridFTP server handles authentication of user• Neill's ESG GridFTp authz callout plugin handles

authorization of data request• ESG/RNI GridFTP DSI module handles data request,

forwards request to ESG BES server running RNI module


13

Client status

RNI Version 0.1 implemented using netCDF version 3 and OPeNDAP’s libnc-dap

In communication with Unidata regarding integration of RNI client with new NetCDF version 4

Developing ncml aggregation in both client and server RNI

Full ESG security support


14


The Remote NetCDF Invocation (RNI)

The client is the netCDF library. It has exactly the same API as the standard C library netCDF, but it can deal with local files or files reachable via HTTP, PPT or gridFTP. The third tier, the BES server can be reached only via PPT. NetCDF services for all NetCDF calls are implemented a a BES module. The middle tier, acts like a proxy between the RNI client and server and deals with security.

15


RNI Architecture

CLIENT DATA

GridFTPOPeNDAP

BES

NetCDFLibrary

RNI Module

connection acts like

RNI Library

16


Characteristics of the RNI as part of a data access system

Full Support of standard OPeNDAP URLs. RNI is being developed with the integrated Unidata/OPeNDAP netCDF library (and CDM)

Transparent access to either standard netCDF files and aggregated datasets via the NetCDF Markup Language (NCML).

For remote containers, all write operations are disable for security. That is, for HTTP/HTTPS, PPT and gridFTP/gsiFTP the RNI system is a read only API.

RNI utilizes Just in Time access. Caching is only for metadata. No pre-fetching of data.

RNI transparently accesses secure (gsiFTP, HTTPS) or insecure (gridFTP, HTTP) remote data.

17


Status of the RPI work (OPeNDAP)

The primary accomplishments for this subproject in the past year has been• The complete request-response for all netCDF API calls has been

completed, that is, the two ends have been developed.• We have highly optimized core components of the standard

OPeNDAP framework to support the performance goals.• We have established the knowledge of how the middle tier (the

proxy) will be incorporated into the complete system.• Our work was presented in AGU fall 2007 and EGU 2008.

So far all of the goals established for the first stage of the project has been completed.

Next stage• integration with product server for gateway and data node data

access• Functional (with enhancements) replacement for TDS

18

Future, will not elaborate

Storage Resource Manager / DMLite as a client Return as RDF Return as PML (Provenance – Proof Markup Language)


Date post:	11-Jan-2016
Category:	Documents
Upload:	chuong
View:	20 times
Download:	0 times