An OPeNDAP server for CORIOLIS data centre

• Context and functions• Architecture and technical description • Planned improvements

T. Loubrieu, A. Forest (IFREMER)

DAP4COR : An OPeNDAP server for CORIOLIS data centre

CORIOLIS is an operational data centre for ocean in-situ measurements.As such CORIOLIS :

Collects measurements for various observation systems (ARGO profilers, measurement from research ships, XBT from Volontary observation ships, moorings, …)Apply various quality controls on datasets.Disseminates datasets as asked by users (ocean forecasting systems, ARGO, Carbocean, …)

The datasets are fully managed in an Oracle RDBMS until they are disseminated as netCDF files in ftp directories or requested through a GUI web interface.

DAP4COR : external usage requirements

Open a standard interface on CORIOLIS database :Enable interactive subsetting of the datasets (even with scripts)Provide an oceanography oriented, high level, interface so that data management and low level information are hidden to the end-user.

Promote CORIOLIS data centre interoperabilityUse DAPPER ocean in-situ vertical profile DAP model.

DAP4COR is not intended to be distributed and installed outside of CORIOLIS data centre.

DAP4COR : internal requirements

Demonstrate a robust and highly capable federating service for dissemination :

Improve reactivity in the development of new dissemination tools.

Facilitate the maintenance of the multiple dissemination tools by making them independant from the low level RDBMS model.

Avoid datasets redundancy (Oracle RDBMS / netCDF files) to ease data update management.

Better resources management by using a unique RDBMS pool of connexion.

Use J2EE architecture (java + tomcat) because of system infrastructure and available skills.

DAP4COR : functions

DAP4COR is Web Application enabling :OPeNDAP request interpretation for ocean in-situ profile dataset subsettingOPeNDAP, DAPPER-like response generation from RDBMS profile data.

DAP4COR Dataset configuration (on SQL criteria on the RDBMS native storage).

DAP4COR command line client for saving extraction as NetCDF ARGO profiles file.

DAP4COR : request syntax example

http://www.ifremer.fr/dap4cor/ server base URL

dap4cor-coriolis dataset

. separator

[dds|das|ascii|dods] output extension

? separator

location.profile.PRES projected variable

, separator

location.profile.TEMP projected variable

& separator

location.TIME>1136505600000.0 selection criteria

& separator

location.TIME<1138579200000.0 selection criteria

& separator

location.attributes.PLATFORM_NUMBER="DACF" selection criteria

DAP4COR : user’s subsetting criterias available

Platform code

Geographical areaTimeVertical reference : depth, pressure

Available measurements

Measurement values

ARGO profiling float

T/S diagramOn a small ocean box

Dissolved oxygenProfiles

for 2 weeks

MediterraneanSalinity profiles

One month

DAP4COR : dataset configuration

Main data (location, time, measurements) are stored in the same 3 tables.Ancillary data are distributed into numerous exotics tables (CORIOLIS « expert » domain).DA4COR Datasets can be configured according to user needs by defining low level SQL criteria on the Oracle DBMS :

Subset of platforms (gliders, elephant seals, moorings array…)Instrument types (CTD, XBT, XCTD, …)Quality assesment level (real time, near real time, delayed mode…)Geographical area (ocean basin, WOCE reference section), by using Oracle Spatial functions (GIS).…

Thanks to that, a most of the RDBMS model complexity is hidden to the user.

The most relevant datasets are configured on server as SQL pre-defined SQL criteria without data redundancy.

DAP4COR : dataset configuration examples

WOCE reference section, for example AX7

Mediterannean seaMeasurements from elephant sealGliders datasetGTS datasets (BATHY, TESAC messages)Sea Surface dataset (for satellite calibration)Pre-processed profiles : pressure/depth conversion, interpolation on standard vertical levels, parameters conversion… To Be Done …

DAP4COR : response format

Ocean in-situ vertical profiles as proposed in PMEL’s DAPPER server :

One sequence of profilesFor each profile :

First level information (time, location + quality control results)Metadata (profile attributes)Sequence of the measured levels (vertical reference value, measured parameters values, quality control information)

Maximum number of vertical profiles in each response : 1000

DAP4COR : command line client (dap4cor2netCDF)

Windows/Unix or Linux portable software.

Uses DAP4COR opendap request as input.

Saves the subset as NetCDF ARGO profile file.

Command line tool which can be used as batch (usefull for operational requirements).

DAP4COR : system architecture

tomcat application server

DAP4COR

JDAP 1.1.7

DAP4CORDatasets

configurationtomcat JDBC

connection pool

Oracle RDBMS

Spatial functions

Components we rely on :JDAP 1.1.7Tomcat application server (enabling load balancing)Tomcat JDBC connection poolOracle RDBMSOracle spatial functions (GIS)

DAP4COR : a 3 layers software architecture

DAP request interpretation/data model generation based on java API (JDAP 1.1.7)

In-situ Vertical Profile business interfaceImplemented in order to manage :

raw profilesOr pre-processed profiles (confidential information hidden, vertical reference conversion, interpolated on standard vertical levels…) To Be Done

Design as a « profile reader » so that memory overflow does not occur. : It reads the lower layer / write the upper layer very frequently so that the smallest possible amount of data is kept is that layer memory.

JDBC convenient resultSets

DAP4COR : profile business interface

Main methods :setConstraint() : initializes the OPeNDAP subsetting criterias (dataset, projected parameters, selection parameters)getStationRoughNumber() – getStationPreciseNumber() : maximum profile quota analysis.Prepare() : initializes the requested profiles subset (before reading)Profiles subset reading methods (mapped on DAPPER sequences order) :

nextStation() : move to next vertical profilegetStationElementValue() : get global information on the current vertical profile (time, location, platform, …)nextLevel() : move to next vertical level for the current profilenextProfileLevelElementValue() : get and move to next measurement for the current vertical level.

Interface implementation is a parameter of the dataset configuration so that we can define, for example :

A dataset of pre-processed profiles with homogeneous vertical references (pressure/depth converted if necessary)…

DAP4COR : planned improvements

Enable big dataset request (thanks to OPeNDAP indexed requests ?)Improve performance especially when only the global profile information (time, location, platform code, …) are requested.Implement alternative profile interface for profile pre-processing (vertical level conversion, interpolation on standard vertical levels…)

Provide a new OpeNDAP interface for other CORIOLIS datasets :

time series at a fixed locationalong track observations

Keep on following the DAPPER observation data model convention and improve compliance with it.

DAP4COR : links

Server URL :http://www.ifremer.fr/dap4cor

Documentation and related tools :ftp://ftp.ifremer.fr/ifremer/coriolis/tools/dap4cor

Containing :FAQ documentJava client which enable saving datasets as NetCDF ARGO profiles file (dap4cor2netcdf).Matlab client script examplesPython client script example