DMAC DeDMAC De--MystifiedMystifiedMacooraMacoora Meeting, November 17, 2009Meeting, November 17, 2009

Eoin (Owen) HowlettEoin (Owen) Howlett

How to speak GEEK…Is your desk a mess?Is your desk a mess? Do you have empty soda cans all around you? Do you shower less than once a week? -- twice a week?

th d ?-- every other day? -- every day? -- multiple times a day?

Whi h d b t d ib i ifi t th (SO)?Which word best describes your significant other (SO)?-I have no SO-Geek / Nerd-Smart-Hunk / Pretty-Dumb-LeftRight-Right

-A Dell

Honestly, do you really have a significant other?I already said no-I already said no

-You caught me-Yes

What is XML?<Name>

</Name>

What is XML?<Name>

M BMacoora Buoys</Name>

What is XML?<name>Macoora Buoys</name><Placemark><Placemark><name>Scarlet Knight</name><ExtendedData><Data name="Buoy ID"><Data name= Buoy ID ><value>1111</value>

</Data><Data name="Temperature">ata a e e pe atu e<value>52</value>

</Data></ExtendedData><Point><coordinates>-73.5,39.5043</coordinates>

</Point> /Pl k</Placemark>

HF Radarada

STPSS S

HF Radar Wavesada a es

GlidersG de s

Satellite-derived Ocean ColorSate te de ed Ocea Co o

Satellite-derived SSTSate te de ed SS

Stevens Model – SalinitySte e s ode Sa ty

Umass Dartmouth ModelU ass a t out ode

Rutgers ROMS Modelutge s O S ode

Aggregate Viewgg egate e

KML Examplesp

http://asascience_pws.s3.amazonaws.com/PWS_DATA.kml

What is a Web Service?

User Request Web Service

User Response

What is a Web Service?

Public Class TempConvert :Inherits WebService

<WebMethod()> Public Function FahrenheitToCelsius(B V l F h h it A St i ) A St i(ByVal Fahrenheit As String) As String

dim fahrfahr=trim(replace(Fahrenheit,",","."))if fahr="" or IsNumeric(fahr)=false then return "Error"

t ((((f h ) 32) / 9) * 5)return ((((fahr) - 32) / 9) * 5)end function

<WebMethod()> Public Function CelsiusToFahrenheit(ByVal Celsius As String) As String(ByVal Celsius As String) As String

dim celcel=trim(replace(Celsius,",","."))if cel="" or IsNumeric(cel)=false then return "Error"return ((((cel) * 9) / 5) + 32)return ((((cel) * 9) / 5) + 32)

end function

end class http://www.w3schools.com/webservic/t tes/tempconvert.asmx

What is WMS?

User Request Web Map Service

Window of interestLayers

Service

yTime of interest

HF radarHF radar

GIS DataGIS DataSIS Data (NetCDF, SIS Data (NetCDF,

GRIB, Other)GRIB, Other)

ArcGIS Server, Geoserver MMS

Metocean WMS ServerGeoserver, MMS,

Other WMSServer

WMS - getCapabiltiesWMS getCapabilties• allows the server to advertise what it can do:

– available layers– supported output projections– supported output formats– scale hints– extent of data

• XML formatXML format

http://onearth.jpl.nasa.gov/wms.cgi?request=GetCapabilities

WMS - getMapg p• allows the retrieval of a map from a web

serverserver• user supplies bounding box, image size,

format error handling etcformat, error handling, etc…• server responds with an “image”, typically a

web-ready format like GIF, PNG or TIFF

http://24 249 210 120/MARCOOS/wms/shake asp?request=http://24.249.210.120/MARCOOS/wms/shake.asp?request=GetMap&service=WMS&version=1.1.1&srs=EPSG:9805&format=image/png&LAYERS=NCOM_SST&BBOX=-79 756 36 12 68 399 42 935&TIME 2009 1179.756,36.12,-68.399,42.935&TIME=2009-11-16T17:00&width=1000&height=600

WMS Examples

KML Google Earth•KML- Google Earth•Servir/Nasa world wind•MacooraMacoora•Arcview

Beta Web Site

OGC Standards that are used

Web Map Service

WMS

Web Coverage Service

WCS

Web Feature Service

WFSKML

WMS WCS WFS

I G id F t F tImage

JPG, PNG, etc.

Grid

GeoTiff, NetCDF etc

Features

XML, etc.

Features

XMLNetCDF, etc.

Request to Server:

•Current Zoom Window

•What Layers

•Current Time

OGC Standards Pros/Cons

Standard Advantages Weaknesses

WMS •Many technologies support WMS•Wide client side support

•Returns a “picture”, no actual data•Symbology and legends are•Wide client side support

•Good for server-side rendering of large datasets

•Symbology and legends are complicated•Inconsistent projection implementation

WCS

WFS

KMLKML

OGC Standards Pros/Cons

Standard Advantages Weaknesses

WMS •Many technologies support WMS•Wide client side support

•Returns a “picture”, no actual data•Symbology and legends are•Wide client side support

•Good for server-side rendering of large datasets

•Symbology and legends are complicated•Inconsistent projection implementation

WCS •Useful for uniform rectangular id

•Not widely supported on servers and/or li tgrids clients

•No support for unstructured and non-uniform grids

WFSWFS

KMLKML

OGC Standards Pros/Cons

Standard Advantages Weaknesses

WMS •Many technologies support WMS•Wide client side support

•Returns a “picture”, no actual data•Symbology and legends are•Wide client side support

•Good for server-side rendering of large datasets

•Symbology and legends are complicated•Inconsistent projection implementation

WCS •Useful for uniform rectangular id

•Not widely supported on servers and/or li tgrids clients

•No support for unstructured and non-uniform grids

WFS •Very flexible •Not practical for large amounts of dataWFS y•Provides all relevant information for data•Allows for custom client-side rendering and analysis

p g•Poor client support, not widely supported•Requires a sophisticated client applicationg y

•Extensiblepp

KML

OGC Standards Pros/ConsSt d d Ad t W kStandard Advantages Weaknesses

WMS •Many technologies support WMS•Wide client side supportG d f id d i f

•Returns a “picture”, no actual data•Symbology and legends are

li d•Good for server-side rendering of large datasets

complicated•Inconsistent projection implementation

WCS •Useful for uniform rectangular grids

•Not widely supported on servers and/or clientsg•No support for unstructured and non-uniform grids

WFS •Very flexible•Provides all relevant information

•Not practical for large amounts of data•Poor client support not widely•Provides all relevant information

for data•Allows for custom client-side rendering and analysis•Extensible

•Poor client support, not widely supported•Requires a sophisticated client application

•Extensible

KML •Very flexible•Provides all relevant information for data

•Not practical for large amounts of data•Creating KML for complex symbols (e.g. rotating vectors) not simple

•Widely used, supported by Google Earth and Google Maps so easy to deploy

What is SOS?User RequestUser Request

•Window of interestSensor Observation

•Observation Platforms

•Parameters

ObservationService

•Parameters

•Time of interest

16T08:30:00Z,0.800,53.000,1.300Wind,urn:x-noaa:def:station:NOAA NOS CO OPS::8516945 "Kingsnoaa:def:station:NOAA.NOS.CO-OPS::8516945, Kings Point, NY",40.8103,-73.7649,2009-11-16T08:36:00Z

Sensor Systems

SOS for in situ Observationsy

C bilti XML

Table ofContents

Observations

Capabilties XML

SOS

Contents

Obs. & Meas. (O&M) XML

SOS(Sensor

ObservationService)

Data ValuesData

ProviderQualityControl

Service)

ProcedureMetadataMetadata

SensorML

SOS GetObservation ResponsepXML Encoding of In-Situ Data

XMLXMLExtensible Markup Language

Generic method for structuring text data

OGC GMLGeography Markup Language

XML that can represent any geospatial feature

specializedby

XML that can represent any geospatial feature

OGC O&MObservations and Measurements Model

specializedby Observations and Measurements Model

GML that describes the act of measuring real-world phenomena and the result of the measurement

33

SOS ExampleSOS Example

NDBC Sit•NDBC Sitehttp://sdf.ndbc.noaa.gov/sosp g

•EDC SOS•EDC - SOS

What is What is NetCDFNetCDF??Network Common Data Format (NetCDF)

Developed at the Unidata Program of the University Corporation for Atmospheric Research (UCAR) (htt // id t d )(UCAR) (http://my.unidata.ucar.edu).

NetCDF has been adopted for use in earth oceanNetCDF has been adopted for use in earth, ocean and atmospheric sciences.

Self-describing format capable of conveying large sets of array-oriented data.

Can be served with OPeNDAP

NetCDFNetCDF ConventionsConventions

COARDS conventionCOARDS conventionCOARDS conventionCOARDS conventionsponsored by the Cooperative Ocean-Atmosphere Research Data Service

CF conventionCF conventionClimate and Forecast Metadata

NetCDF SchemasNetCDF Schemas

Rectangular Grid

Sample CDL:netcdf sample_rectangular_grid_no_z {dimensions:

time = UNLIMITED ; // (10 currently)x = 214 ;y = 215 ;Rectangular Grid

Multi point geo static non

variables:float time(time) ;

time:long_name = "time" ;time:units = "minutes since 1979-12-31" ;

float lon(y, x) ;lon:long_name = "latitude" ;lon:units = "degrees north" ;Multi-point, geo-static, non-

gridded cell datalon:units = degrees_north ;

float lat(y, x) ;lat:long_name = "longitude" ;lat:units = "degrees_east" ;

float wind_u(time, ncell) ;wind_u:long_name = "eastward_wind_velocity" ;wind_u:units = "meters/second" ;

Boundary fitted, finite element, variable mesh

_ ;wind_u:missing_value = "99999" ;

float wind_v(time, ncell) ;wind_v:long_name = "northward_wind_velocity" ;wind_v:units = "meters/second" ;wind_v:missing_value = "99999" ;

// global attributes:titl "S l fil f t l id "element, variable mesh

Multi-point temporal non-

:title = "Sample nc file for rectangular grid." ;:default_view = "wind_u,wind_v" ;:netcdf_class = "3" ;:netcdf_class_description = "Rectangular grid" ;:reference = "Created: 2004-12-08 10:37:12" ;:time_var = "time" ;:lat var = "lat" ;Multi point, temporal, non

gridded:lat_var lat ;:lon_var = "lon" ;:view_style = "1" ;:cell_size_x = "1.0" ;:cell_size_y = "1.0" ;:Conventions = "CF-1.0" ;

Things got more complicatedThings got more complicated

ApplicationApplication

C

Fortr

C++

Matl

Per

Pyth

Rub

Java

netCDF C library /

C ran

API

+

API

lab

API

lAPI

hon

API

by

API

netCDF Java

a

netCDF C library / API

netCDF Java library / API

tCDF 3 fil OPeNDAP datatCDF 4 fil netCDF-3 file OPeNDAP datanetCDF-4 file

NetCDFNetCDF DemoDemo

•ToolsUI - Local NetCDF file

What is OPeNDAP

netCDF HDF4 HDF5 DSP Tables GRIB FITS CDFFlat

Binary CEDARESML

Data Data Data Data Data Data DataData

netCDF HDF4 HDF5

Data

DSP

Data

Tables GRIB FITS CDF Binary CEDAR

Data

ESML

HDF5

HDF4

FreeFormFITS

CDF CEDAR

netCDF

DSP

JGOFSGeneral

http://tashtego.marine.rutgers.edu:8080/thredds/dodsC/cool/codar/totals/macoora8km htmlar/totals/macoora8km.html

OPeNDAP Servers1) AVISO SSH: http://www.aviso.oceanobs.com/en/data/data-access-services/opendap/opendap-sla-products/index.html2) GHRSST:http //dods jpl nasa go /dods bin/nphhttp://dods.jpl.nasa.gov/dods-bin/nph-hdf/pub/sea_surface_temperature/GHRSST/data/3) Bathymetry: http://coast-enviro.er.usgs.gov/thredds/bathy_catalog.html4) IPCC: http://www2-pcmdi.llnl.gov/esg_data_portal/dapserver/) p p g g p p5) CMAR Various: http://www.marine.csiro.au/remotesensing/restricted/un001.html6) MBARI Various: http://dods.mbari.org/cgi-bin/nph-nc/data/7) NDBC Mooring data: http://dods.ndbc.noaa.gov:8080/opendap//8) Ocean Color/SSH/Rain/Other:8) Ocean Color/SSH/Rain/Other:http://satdat1.gso.uri.edu/opendap/data_collections/table.html9) Satellite fields:http://satdat1.gso.uri.edu/opendap/REASoN/REASoN_DataPortalAccess.html10) TOVS: http://disc1.sci.gsfc.nasa.gov/opendap/tovs/contents.html11) PMEL many ocean data sets:http://ferret.pmel.noaa.gov/thredds/dodsC/data/PMEL/12) JPL many data sets: http://dods jpl nasa gov/12) JPL many data sets: http://dods.jpl.nasa.gov/13) APDRC many data sets: http://apdrc.soest.hawaii.edu/w_data/alldata3.htm

What is THREDDS?THematic Real-time Environmental Distributed Data Services

HTTP Tomcat Server catalog.xmlTHREDDS Server •WCS

•OPeNDAP

NetCDF-Java Application

•HTTPServer

•NetcdfSubset

library

configCatalog.xml

Datasets

motherlode.ucar.edu

EDC Demo

Data Data Data Data Data Data DataData

netCDF HDF4 HDF5

Data

DSP

Data

Tables GRIB FITS CDFFlat

Binary CEDAR

Data

ESML

HDF5 FreeFormFITSnetCDF

DSP

JGOFSGeneral

HDF4 CDF CEDARDSP

HTTP Tomcat Servercatalog.xml

THREDDS Server

•HTTPServer

•WCS•OPeNDAP

NetCDF-Javalibrary

Application•NetcdfSubset

configCatalog.xml

NCML Capabilities:

• Aggregation in time and space using NCMLAggregation in time and space using NCML<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">

<aggregation dimName="time" type="joinExisting"><netcdf location="file:C:\path\to\my1.grib2" /><netcdf location="file:C:\path\to\my2.grib2" /><netcdf location="file:C:\path\to\my3.grib2" />

</aggregation></ t df></netcdf>

• Combine files (separate grid and data?)• Add, edit or rename variables and attributes• Add or rename dimensions• NCML can ‘fix’ broken metadata

E d U A li ti•End User Applications

•SAROPS•Inundation Tools•Inundation Tools•OILMAPL l T t•Larval Transport

SLOSH with IOOS Observations

OILMAP Demo

M th M th M th MonthIntegration

Months 0-12

Month 18 Transport

Month 24 Test &

Evaluation

Month 36

Benchmarked

Salinity

Temp DATA

NWS NDBC

OGC

CO-OPSE. GOMHABS

BULLETIN

ColorCurrentsWinds

Metrics •Efficiency•SpeedSalinity

Currents

L l

A

CON

NDBC

NOS

C

WFS

PACOOSIEA

All 7

Speed•Resolution•AccuracyL thLevel

Color

TENT

CO-OPS

NESDIS

-WCS-

NHCSLOSH

CI MODEL

WindsWavesLevel

•Length•Economics

Winds

Waves

STD

NESDIS Coastwatch

WMS

GML/CSML & NetCDF

HIFPROGRAM

TempWindsWaves

Feb 2008 Aug 2008 Feb 2009 Feb 2010scheduled

actualMay 2008 Aug 2008 Feb 2009 Feb 2010

MARCOOS 7 Core VariablesCOOS Co e a ab es

Moving Towards National DMAC

1. Standardized Access 1. Standardized Access 7. System Monitoring Client7. System Monitoring Client

12 DMAC Functional Components12 DMAC Functional Components

ServicesServices

2. Registry of Services2. Registry of Services

y gy g

8. Format Conversion Service8. Format Conversion Service

9 Data Subscription and9 Data Subscription and3. Mapping & Visualization3. Mapping & Visualization

ServicesServices

9. Data Subscription and 9. Data Subscription and Alert ServicesAlert Services

10. Data Archiving and 10. Data Archiving and 4. IT Security Plan and 4. IT Security Plan and

ValidationValidation

ggRetrieval ServicesRetrieval Services

11. Data Integration Services11. Data Integration Services5. Metadata Profiles5. Metadata Profiles

6. Catalog for Data Discovery6. Catalog for Data Discovery12. Quality Control/Quality 12. Quality Control/Quality

Assurance ProtocolsAssurance Protocols

IOOS Regional Coordination Workshop – August 2009

•WMS (Web Mapping Service)•WFS (Web Feature Service) •WCS (Web Coverage Service)•SOS (Sensor Observation Service)•FGDC (Federal Geographic Data Committee)•GCMD (Global Change Master Directory)•OGC (Open Geospatial Consortium)•TDS (Thredds Data Server)•DAP (Data Access Protocol)•Soap (Simple Object Access Protocol)•XML (Extensible Markup Language)•XSL (Extensible Stylesheet Language)•XSLT (XSL: Transformations)

Conclusions•RA’s and IOOS office are actively working together to implement standards

•Provide tools and “glue” to integrate legacy systems as opposed to redesigningsystems as opposed to redesigning components

Focus on meeting user’s needs•Focus on meeting user s needs

•Sesame Street Generation- Has to be FAST.

•We need to leverage Web 2.0 technologies

Thanks toa s to

David Stuebe, Kelly Knee, Lauren Decker, Kyle WilWilcox

Rich SignellRich Signell

Jeff de La Beaujardière, Rob Ragsdalej , g

John Caron

Ted Haberman

Questions and DiscussionQuestions and Discussion