+ All Categories
Home > Documents > A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet...

A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet...

Date post: 25-Sep-2020
Category:
Upload: others
View: 4 times
Download: 0 times
Share this document with a friend
42
A Smartmet Server based WFS and WCS implementation Finnish Meteorological Institute Mikko Visa, Roope Tervo
Transcript
Page 1: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

A Smartmet Server based WFS and WCS implementation

Finnish Meteorological Institute

Mikko Visa, Roope Tervo

Page 2: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

Finnish Meteorological Institute opened its data in 2013. Basically everything that FMI has property rights was opened. Data is provided freely in machine readable format.

23/09/16 2

FMI Open Data

https://en.ilmatieteenlaitos.fi/open-data

Page 3: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

Introduction

23/09/16 A Smartmet Server based WCS implementation

Roope Tervo, Mikko Visa

3

Page 4: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

• WFS and “WCS-like” download service implementations in production • WCS implementation currently work in progress

23/09/16

Serving Coverage Data in FMI Open Data Portal Roope Tervo, Mikko Visa

4

Implementations at FMI

Page 5: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

WFS implementation

23/09/16 Serving Coverage Data in FMI Open Data Portal

Roope Tervo, Mikko Visa

5

Page 6: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

• Web Feature Service (WFS) 2.0 Simple Profile • Based on stored queries

• Pre-defined data sets with possibility for additional parameters (ie. time and area)

• In-house production, based on Smartmet Server • Currently being opensourced (https://github.com/fmidev)

• In production use since 04/2013 • Observations and point forecasts in GML format

• Grid Series Observation/Point Time Series Observation/Simple Feature

• Gridded data in appropriate format • WFS members contain the metadata ‘envelope’ with a link to the actual data • GRIB1/2 for weather models, NetCDF for sea models, GeoTIFF for radar images

23/09/16

Serving Coverage Data in FMI Open Data Portal Roope Tervo, Mikko Visa

6

WFS download service

Page 7: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

Producing INSPIRE Data Products Point Forecasts

23/09/16 7

File System

Smartmet Server

Memory mapped data.

Server provides logic for

interpolating the data for

requested area and time.

Smartmet Server WFS

Plugin

Page 8: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

Producing INSPIRE Data Products Grid Forecasts 1/2

File System

Smartmet Server

Server provides relevant meta

data to fetch the data content

Smartmet Server WFS

Plugin

23/09/16 8

Page 9: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

Producing INSPIRE Data Products Grid Forecasts 2/2

File System

Smartmet Server

Smartmet Server

Download Plugin

Memory mapped data.

Server provides logic for

interpolating the data for

requested area and time.

23/09/16 9

Page 10: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

•  Provides grid data as binary data •  Supported output formats: GRIB1, GRIB2, NetCDF and Querydata •  Supports all proj.4 projections (depends on output format support) •  Supports slicing by

•  area (bbox)

•  elevation (pressure and/or model level)

•  time (start time, end time and origin time) •  Possibility to define grid resolution by

•  selecting every Nth grid point to x and y direction

•  grid size à data is interpolated to new grid points 23/09/16

Serving Coverage Data in FMI Open Data Portal Roope Tervo, Mikko Visa

10

WCS-like download service

Page 11: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

GetCapabilities: •  Describes capabilities of the service •  Normally start point for client applications •  http://data.fmi.fi/fmi-apikey/fmi-apikey/…/wfs?

request=GetCapabilities

23/09/16 FMI Open Meteorological Data Services | Roope Tervo 11

WFS Basics observations and point forecasts

Page 12: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

DescribeStoredQueries •  List and describes available stored queries (pre-defined

data sets) •  Stored query name and description

•  Possible additional parameters available for the stored query

•  http://en.ilmatieteenlaitos.fi/open-data-manual-fmi-wfs-services

23/09/16 FMI Open Meteorological Data Services | Roope Tervo 12

WFS Basics observations and point forecasts

Page 13: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

GetFeature •  Actual data query •  In FMI service, stored query id is mandatory

•  &storedquery_id=..

•  Other possible parameters depend on the stored query •  Possible values can be fetched with DescribeStoredQueries

request.

•  Location has to be often defined somehow.

http://data.fmi.fi/fmi-apikey/.../wfs?request=GetFeature&storedquery_id=fmi::observations::weather::cities::timevaluepair&bbox=21,62,22,64

23/09/16 FMI Open Meteorological Data Services | Roope Tervo 13

WFS Basics observations and point forecasts

Page 14: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

Location parameters in stored queries &  bbox à area

•  By default, in same CRS with the (requested) CRS. Specify as fifth element if needed (i.e. &bbox=132249,6433579,631999,6933329,epsg:3067)

&  fmisid à FMI id for observation station &  wmo à WMO id for observation station

23/09/16 FMI Open Meteorological Data Services | Roope Tervo 14

WFS Basics observations and point forecasts

Page 15: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

Location parameters in stored queries &  geoid à http://geonames.org id for location &  place à human readable name

•  use region to accurate your query (i.e. &place=kumpula,tampere)

•  can be used several times in one query (i.e. &place=kumpula,helsinki&place=heinola)

&  latlon à coordinates (used only in marine data sets)

23/09/16 FMI Open Meteorological Data Services | Roope Tervo 15

WFS Basics observations and point forecasts

Page 16: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

GetPropertyValue •  Responses only requested part of the GetFeature response •  Use ValueReference=xpath to define part of interest •  Same additional parameters for stored query with GetFeature

request are still valid •  For example:

http://data.fmi.fi/fmi-apikey/…/wfs?request=GetPropertyValue&storedquery_id=fmi::observations::weather::multipointcoverage&place=kumpula,tampere&ValueReference=wfs:FeatureCollection/wfs:member/omso:GridSeriesObservation/om:result

23/09/16 FMI Open Meteorological Data Services | Roope

Tervo 16

WFS Basics observations and point forecasts

Page 17: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

•  WFS response contains the same meta data information as in observations and point forecasts

•  om:result contains gmlcov:RectifiedGridCoverage •  Basically the same with Grid Series Observation

•  domainSet defines the grid (now as regular grid)

•  rangeSet contains data as an external link to the binary content •  rangeType defines the parameters

23/09/16 FMI Open Meteorological Data Services | Roope Tervo 17

WFS basics binary data

Page 18: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

•  Note that fileReference points often to an other service •  Can also be used directly, but

•  It is always good practice to consult WFS for available times and parameters

23/09/16 FMI Open Meteorological Data Services | Roope Tervo 18

WFS basics binary data

Client

WFS

Geoserver WMS

Smartmet Server

Available radar images? Available weather model outputs?

Page 19: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

•  Area, time and (weather) parameters may be defined in the request

•  For weather models, intersection of requested area and available data area is returned

•  For radar images, all images which intersects requested spatial and temporal space are returned

23/09/16 FMI Open Meteorological Data Services | Roope Tervo 19

WFS basics binary data

Requested area

Radar images

Radar images Requested area

Model coverage

Weather models

Page 20: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

Radar images •  The reference points to a “original” gray scale GeoTiff image •  Images can also be downloaded as color images

•  Remove &styles=raster from the data request

•  But then information is lost

•  Used SLD-files can be downloaded from https://github.com/fmidev/opendata-resources/tree/master/sld

•  Consult om:parameter element for single radar measurement parameters

•  Scanning angle, bin count and bin length

23/09/16 FMI Open Meteorological Data Services | Roope Tervo 20

WFS basics binary data

Page 21: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

Numerical models •  The reference points to (often a subset) the model output in

appropriate format •  GRIB1/2 for weather models

•  NetCDF for marine models

•  Note that whole model output can be large •  Up to 17 GB

•  Tip: download always only area, parameters and time range of interest

23/09/16 FMI Open Meteorological Data Services | Roope Tervo 21

WFS basics binary data

Page 22: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

WCS implementation (in progress)

23/09/16 Serving Coverage Data in FMI Open Data Portal

Roope Tervo, Mikko Visa

22

Page 23: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

• Well proven existing data server (Smartmet Server) • Currently being opensourced! (https://github.com/fmidev)

• WFS implementation already based on Smartmet Server in production since 04/2013 • Why not use already existing servers

• Geoserver WCS only returns multipart format,GRIB/NetCDF support missing • More difficult integration to FMI’s own production system

23/09/16

Serving Coverage Data in FMI Open Data Portal Roope Tervo, Mikko Visa

23

Why build or own WCS implementation?

Page 24: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

• Version 2.0.0 and 2.0.1 • GetCapabilities

• INSPIRE metadata included • DescribeCoverage

• validtime, area, level • GetCoverage

• Whole data in native CRS (RectifiedGridCoverage) or • outputCrs=http://www.opengis.net/def/crs/EPSG/0/XXXX

• Slice or trim: subset (x,y,z,t) using KVP/GET or XML/POST • Time can be given in ISO or unixtime format

• Formats: NetCDF partially implemented, GRIB1/2 coming later • format=application/netcdf

23/09/16

Serving Coverage Data in FMI Open Data Portal Roope Tervo, Mikko Visa

24

Supports

Page 25: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

• HIRLAM model temperature trimmed to selected area, output in NetCDF / WGS84

• http://beta.fmi.fi/wcs?service=WCS&request=GetCoverage&coverageid=hirlam-temperature&format=application/netcdf&subset=x(-30,30)&subset=y(60,90)&outputCrs=http://www.opengis.net/def/crs/EPSG/0/4326

23/09/16

Serving Coverage Data in FMI Open Data Portal Roope Tervo, Mikko Visa

25

Examples (WCS)

Page 26: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

• HBM salinity, one level, one timestep

• http://beta.fmi.fi/wcs?service=WCS&request=getCoverage&coverageid=hbm-salinity&output=application/netcdf&subset=z(400)&subset=t(“2016-09-15T13:00:00Z”)

• HBM salinity, one level, time interval • http://beta.fmi.fi/wcs?service=WCS&request=getCoverage&coverageid=hbm-salinity&format=application/netcdf&subset=z(400)&subset=t(“2016-09-16T13:00:00Z”,”2016-09-16T14:00:00Z”)

23/09/16

Serving Coverage Data in FMI Open Data Portal Roope Tervo, Mikko Visa

26

Examples (WCS)

Page 27: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

• HBM salinity, one level, 2 timesteps, single point, • http://beta.fmi.fi/wcs?service=WCS&request=getCoverage&coverageid=hbm-salinity&format=application/netcdf&subset=x(24.937592)&subset=y(59.858954)&subset=z(10)&subset=t(“2016-09-16T13:00:00Z”,”2016-09-16T14:00:00Z”)

• [[email protected] ~]$ ncdump /tmp/x.nc • netcdf x { • dimensions: •  time = 2 ; •  level = 1 ; •  yc = 1 ; •  xc = 1 ;

23/09/16

Serving Coverage Data in FMI Open Data Portal Roope Tervo, Mikko Visa

27

Examples (WCS)

Page 28: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

• No support for MetOcean WCS 2.0 Profile yet • MetOcean WCS 2.0 Profile status?

• WCS 2.1 (including CIS 1.1) • GRIB output

23/09/16

Serving Coverage Data in FMI Open Data Portal Roope Tervo, Mikko Visa

28

Still to implement (WCS)

Page 29: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

• Internal use: • Replace the “wcs-like” download plugin

• External use • Add to Open Data / INSPIRE interface as replacement for current non-standard “wcs-like” download plugin

23/09/16

Serving Coverage Data in FMI Open Data Portal Roope Tervo, Mikko Visa

29

Usage planned for WCS

Page 30: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

Implementation challenges

23/09/16 Serving Coverage Data in FMI Open Data Portal

Roope Tervo, Mikko Visa

30

Page 31: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

• CRS/SRS challenges • many weather models are calculated in rotated lat-lon • Lat/lon order dependent on CRS/SRS, confusing..!

• How to define time interval: starttime,endtime or starttime/endtime? • How to define a dataset?

• In INSPIRE context often a new version, but this is not feasible in very frequently updated data such as weather data

23/09/16

Serving Coverage Data in FMI Open Data Portal Roope Tervo, Mikko Visa

31

Challenges 1/2

Page 32: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

• Reprojection not well supported in WCS, however there is often need for this in the MetOcean domain • Lack of WCS 2.x clients, hard to test!

• Clients supporting 1.x exist (QGIS for example) • You have the web browser of course..

• Dimensions • Now x, y, z, t… • Currently only latest model run is offered, one solution would be to put also this in the CoverageId • How to handle different (meteorological) parameters in model data? Currently need to be different coverages..

• What about EPS, probability forecasts…

23/09/16

Serving Coverage Data in FMI Open Data Portal Roope Tervo, Mikko Visa

32

Challenges 2/2

Page 33: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

Experiences

23/09/16 Serving Coverage Data in FMI Open Data Portal

Roope Tervo, Mikko Visa

33

Page 34: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

• Data is often 5 dimensional with irregular grid and

time intervals

Multidimensional data is hard to

handle

Hard Parts

23/09/16 34

Page 35: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

Names, units, levels, time intervals… Handling parameters

is one of the hardest parts.

Hard Parts

23/09/16 35

Page 36: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

Converting everything to one

data format is complicated, time consuming and

expensive.

Multiple data formats are required

Hard Parts

23/09/16 36

Page 37: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

And a little over 440 000 data downloads

per day (5,2 req/s)

At the moment about 10000

registered users

Some Experiences

23/09/16 37

Page 38: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

INSPIRE is a long project. Better to look forward than

backward

Although standards are followed, there’s

a gap between provided data model

and clients’ capabilities

Some Experiences

23/09/16 38

Page 39: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

There’s not always support for HTTP

1.1. (in clients, proxies, load balancers…)

Data need to be transferred in

chunked encoding (requires HTTP 1.1)

Hard Parts

23/09/16 39

Page 40: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

For example many weather models are calculated in rotated

lat-lon. Supporting several projections for the

data is vital.

Lessons learned

23/09/16 40

Page 41: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

23/09/16 41

Data Models Popularity Comparison

80

19.8

0.2 0

10 20 30 40 50 60 70 80 90

Downloads[%]

Page 42: A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet •MetOcean WCS 2.0 Profile status? •WCS 2.1 (including CIS 1.1) •GRIB output

www.fmi.fi

http://www.slideshare.net/tervo/ https://en.ilmatieteenlaitos.fi/open-data


Recommended