Home >Documents >A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet...

A Smartmet Server based WFS and WCS implementation...•No support for MetOcean WCS 2.0 Profile yet...

Date post:25-Sep-2020
Category:
View:1 times
Download:0 times
Share this document with a friend
Transcript:
  • A Smartmet Server based WFS and WCS implementation

    Finnish Meteorological Institute

    Mikko Visa, Roope Tervo

  • Finnish Meteorological Institute opened its data in 2013. Basically everything that FMI has property rights was opened. Data is provided freely in machine readable format.

    23/09/16 2

    FMI Open Data

    https://en.ilmatieteenlaitos.fi/open-data

  • Introduction

    23/09/16 A Smartmet Server based WCS implementation

    Roope Tervo, Mikko Visa

    3

  • • WFS and “WCS-like” download service implementations in production • WCS implementation currently work in progress

    23/09/16

    Serving Coverage Data in FMI Open Data Portal Roope Tervo, Mikko Visa

    4

    Implementations at FMI

  • WFS implementation

    23/09/16 Serving Coverage Data in FMI Open Data Portal

    Roope Tervo, Mikko Visa

    5

  • • Web Feature Service (WFS) 2.0 Simple Profile • Based on stored queries

    • Pre-defined data sets with possibility for additional parameters (ie. time and area)

    • In-house production, based on Smartmet Server • Currently being opensourced (https://github.com/fmidev)

    • In production use since 04/2013 • Observations and point forecasts in GML format

    • Grid Series Observation/Point Time Series Observation/Simple Feature

    • Gridded data in appropriate format • WFS members contain the metadata ‘envelope’ with a link to the actual data • GRIB1/2 for weather models, NetCDF for sea models, GeoTIFF for radar images

    23/09/16

    Serving Coverage Data in FMI Open Data Portal Roope Tervo, Mikko Visa

    6

    WFS download service

  • Producing INSPIRE Data Products Point Forecasts

    23/09/16 7

    File System

    Smartmet Server

    Memory m

    apped dat

    a.

    Server pr

    ovides log

    ic for

    interpolatin

    g the data

    for

    requested

    area and t

    ime.

    Smartmet Server WFS

    Plugin

  • Producing INSPIRE Data Products Grid Forecasts 1/2

    File System

    Smartmet Server

    Server pr

    ovides rele

    vant meta

    data to fet

    ch the dat

    a content

    Smartmet Server WFS

    Plugin

    23/09/16 8

  • Producing INSPIRE Data Products Grid Forecasts 2/2

    File System

    Smartmet Server

    Smartmet Server

    Download Plugin

    Memory m

    apped dat

    a.

    Server pr

    ovides log

    ic for

    interpolatin

    g the data

    for

    requested

    area and t

    ime.

    23/09/16 9

  • •  Provides grid data as binary data •  Supported output formats: GRIB1, GRIB2, NetCDF and Querydata •  Supports all proj.4 projections (depends on output format support) •  Supports slicing by

    •  area (bbox)

    •  elevation (pressure and/or model level)

    •  time (start time, end time and origin time) •  Possibility to define grid resolution by

    •  selecting every Nth grid point to x and y direction

    •  grid size à data is interpolated to new grid points 23/09/16

    Serving Coverage Data in FMI Open Data Portal Roope Tervo, Mikko Visa

    10

    WCS-like download service

  • GetCapabilities: •  Describes capabilities of the service •  Normally start point for client applications •  http://data.fmi.fi/fmi-apikey/fmi-apikey/…/wfs?

    request=GetCapabilities

    23/09/16 FMI Open Meteorological Data Services | Roope Tervo 11

    WFS Basics observations and point forecasts

  • DescribeStoredQueries •  List and describes available stored queries (pre-defined

    data sets) •  Stored query name and description •  Possible additional parameters available for the stored query

    •  http://en.ilmatieteenlaitos.fi/open-data-manual-fmi-wfs-services

    23/09/16 FMI Open Meteorological Data Services | Roope Tervo 12

    WFS Basics observations and point forecasts

  • GetFeature •  Actual data query •  In FMI service, stored query id is mandatory

    •  &storedquery_id=.. •  Other possible parameters depend on the stored query

    •  Possible values can be fetched with DescribeStoredQueries request.

    •  Location has to be often defined somehow.

    http://data.fmi.fi/fmi-apikey/.../wfs?request=GetFeature&storedquery_id=fmi::observations::weather::cities::timevaluepair&bbox=21,62,22,64

    23/09/16 FMI Open Meteorological Data Services | Roope Tervo 13

    WFS Basics observations and point forecasts

  • Location parameters in stored queries &  bbox à area

    •  By default, in same CRS with the (requested) CRS. Specify as fifth element if needed (i.e. &bbox=132249,6433579,631999,6933329,epsg:3067)

    &  fmisid à FMI id for observation station &  wmo à WMO id for observation station

    23/09/16 FMI Open Meteorological Data Services | Roope Tervo 14

    WFS Basics observations and point forecasts

  • Location parameters in stored queries &  geoid à http://geonames.org id for location &  place à human readable name

    •  use region to accurate your query (i.e. &place=kumpula,tampere)

    •  can be used several times in one query (i.e. &place=kumpula,helsinki&place=heinola)

    &  latlon à coordinates (used only in marine data sets)

    23/09/16 FMI Open Meteorological Data Services | Roope Tervo 15

    WFS Basics observations and point forecasts

  • GetPropertyValue •  Responses only requested part of the GetFeature response •  Use ValueReference=xpath to define part of interest •  Same additional parameters for stored query with GetFeature

    request are still valid •  For example:

    http://data.fmi.fi/fmi-apikey/…/wfs?request=GetPropertyValue&storedquery_id=fmi::observations::weather::multipointcoverage&place=kumpula,tampere&ValueReference=wfs:FeatureCollection/wfs:member/omso:GridSeriesObservation/om:result

    23/09/16 FMI Open Meteorological Data Services | Roope Tervo 16

    WFS Basics observations and point forecasts

  • •  WFS response contains the same meta data information as in observations and point forecasts

    •  om:result contains gmlcov:RectifiedGridCoverage •  Basically the same with Grid Series Observation

    •  domainSet defines the grid (now as regular grid) •  rangeSet contains data as an external link to the binary content •  rangeType defines the parameters

    23/09/16 FMI Open Meteorological Data Services | Roope Tervo 17

    WFS basics binary data

  • •  Note that fileReference points often to an other service •  Can also be used directly, but

    •  It is always good practice to consult WFS for available times and parameters

    23/09/16 FMI Open Meteorological Data Services | Roope Tervo 18

    WFS basics binary data

    Client

    WFS

    Geoserver WMS

    Smartmet Server

    Available radar images? Available weather model outputs?

  • •  Area, time and (weather) parameters may be defined in the request

    •  For weather models, intersection of requested area and available data area is returned

    •  For radar images, all images which intersects requested spatial and temporal space are returned

    23/09/16 FMI Open Meteorological Data Services | Roope Tervo 19

    WFS basics binary data

    Requested area

    Radar images

    Radar images Requested area

    Model coverage

    Weather models

  • Radar images •  The reference points to a “original” gray scale GeoTiff image •  Images can also be downloaded as color images

    •  Remove &styles=raster from the data request •  But then information is lost •  Used SLD-files can be downloaded from

    https://github.com/fmidev/opendata-resources/tree/master/sld •  Consult om:parameter element for single radar measurement

    parameters •  Scanning angle, bin count and bin length

    23/09/16 FMI Open Meteorological Data Services | Roope Tervo 20

    WFS basics binary data

  • Numerical models •  The reference points to (often a subset) the model output in

    appropriate format •  GRIB1/2 for weather models •  NetCDF for marine models

    •  Note that whole model output can be large •  Up to 17 GB •  Tip: download always only area, parameters and time range of

    interest

    23/09/16 FMI Open Meteorological Data Services | Roope Tervo 21

    WFS basics binary data

  • WCS implementation (in progress)

    23/09/16 Serving Coverage Data in FMI Open Data Portal

    Roope Tervo, Mikko Visa

    22

  • • Well proven existing data server (Smartmet Server) • Currently being opensourced! (https://github.com/fmidev)

    • WFS implementation already based on Smartmet Server in production since 04/2013 • Why not use already existing servers

    • Geoserver WCS only returns multipart format,GRIB/NetCDF support missing • More difficult integration to FMI’s own production system

    23/09/16

    Serving Coverage Data in FMI Open Data Portal Roope Tervo, Mikko Visa

    23

    Why build or own WCS implementation?

  • • Version 2.0.0 and 2.0.1 • GetCapabilities

    • INSPIRE metadata included • DescribeCoverage

    • validtime, area, level • GetCoverage

    • Whole data in native CRS (RectifiedGridCoverage) or • outputCrs=http://www.opengis.net/def/crs/EPSG/0/XXXX

    • Slice or trim: subset (x,y,z,t) using KVP/GET or XML/POST • Time can be given in ISO or unixtime format

    • Formats: NetCDF partially implemented, GRIB1/2 coming later • format=application/netcdf

    23/09/16

    Serving Coverage Data in FMI Open Data Portal Roope Tervo, Mikko Visa

    24

    Supports

  • • HIRLAM model temperature trimmed to selected area, output in NetCDF / WGS84

    • http://beta.fmi.fi/wcs?service=WCS&request=GetCoverage&coverageid=hirlam-temperature&format=application/netcdf&subset=x(-30,30)&subset=y(60,90)&outputCrs=http://www.opengis.net/def/crs/EPSG/0/4326

    23/09/16

    Serving Coverage Data in FMI Open Data Portal Roope Tervo, Mikko Visa

    25

    Examples (WCS)

  • • HBM salinity, one level, one timestep

    • http://beta.fmi.fi/wcs?service=WCS&request=getCoverage&coverageid=hbm-salinity&output=application/netcdf&subset=z(400)&subset=t(“2016-09-15T13:00:00Z”)

    • HBM salinity, one level, time interval • http://beta.fmi.fi/wcs?service=WCS&request=getCoverage&coverageid=hbm-salinity&format=application/netcdf&subset=z(400)&subset=t(“2016-09-16T13:00:00Z”,”2016-09-16T14:00:00Z”)

    23/09/16

    Serving Coverage Data in FMI Open Data Portal Roope Tervo, Mikko Visa

    26

    Examples (WCS)

  • • HBM salinity, one level, 2 timesteps, single point, • http://beta.fmi.fi/wcs?service=WCS&request=getCoverage&coverageid=hbm-salinity&format=application/netcdf&subset=x(24.937592)&subset=y(59.858954)&subset=z(10)&subset=t(“2016-09-16T13:00:00Z”,”2016-09-16T14:00:00Z”)

    • [[email protected] ~]$ ncdump /tmp/x.nc • netcdf x { • dimensions: •  time = 2 ; •  level = 1 ; •  yc = 1 ; •  xc = 1 ;

    23/09/16

    Serving Coverage Data in FMI Open Data Portal Roope Tervo, Mikko Visa

    27

    Examples (WCS)

  • • No support for MetOcean WCS 2.0 Profile yet • MetOcean WCS 2.0 Profile status?

    • WCS 2.1 (including CIS 1.1) • GRIB output

    23/09/16

    Serving Coverage Data in FMI Open Data Portal Roope Tervo, Mikko Visa

    28

    Still to implement (WCS)

  • • Internal use: • Replace the “wcs-like” download plugin

    • External use • Add to Open Data / INSPIRE interface as replacement for current non-standard “wcs-like” download plugin

    23/09/16

    Serving Coverage Data in FMI Open Data Portal Roope Tervo, Mikko Visa

    29

    Usage planned for WCS

  • Implementation challenges

    23/09/16 Serving Coverage Data in FMI Open Data Portal

    Roope Tervo, Mikko Visa

    30

  • • CRS/SRS challenges • many weather models are calculated in rotated lat-lon • Lat/lon order dependent on CRS/SRS, confusing..!

    • How to define time interval: starttime,endtime or starttime/endtime? • How to define a dataset?

    • In INSPIRE context often a new version, but this is not feasible in very frequently updated data such as weather data

    23/09/16

    Serving Coverage Data in FMI Open Data Portal Roope Tervo, Mikko Visa

    31

    Challenges 1/2

  • • Reprojection not well supported in WCS, however there is often need for this in the MetOcean domain • Lack of WCS 2.x clients, hard to test!

    • Clients supporting 1.x exist (QGIS for example) • You have the web browser of course..

    • Dimensions • Now x, y, z, t… • Currently only latest model run is offered, one solution would be to put also this in the CoverageId • How to handle different (meteorological) parameters in model data? Currently need to be different coverages..

    • What about EPS, probability forecasts…

    23/09/16

    Serving Coverage Data in FMI Open Data Portal Roope Tervo, Mikko Visa

    32

    Challenges 2/2

  • Experiences

    23/09/16 Serving Coverage Data in FMI Open Data Portal

    Roope Tervo, Mikko Visa

    33

  • • Data is often 5 dimensional with irregular grid and

    time intervals

    Multidimensional data is hard to

    handle

    Hard Parts

    23/09/16 34

  • Names, units, levels, time intervals… Handling parameters

    is one of the hardest parts.

    Hard Parts

    23/09/16 35

  • Converting everything to one

    data format is complicated, time consuming and

    expensive.

    Multiple data formats are required

    Hard Parts

    23/09/16 36

  • And a little over 440 000 data downloads

    per day (5,2 req/s)

    At the moment about 10000

    registered users

    Some Experiences

    23/09/16 37

  • INSPIRE is a long project. Better to look forward than

    backward

    Although standards are followed, there’s

    a gap between provided data model

    and clients’ capabilities

    Some Experiences

    23/09/16 38

  • There’s not always support for HTTP

    1.1. (in clients, proxies, load balancers…)

    Data need to be transferred in

    chunked encoding (requires HTTP 1.1)

    Hard Parts

    23/09/16 39

  • For example many weather models are calculated in rotated

    lat-lon. Supporting several projections for the

    data is vital.

    Lessons learned

    23/09/16 40

  • 23/09/16 41

    Data Models Popularity Comparison

    80

    19.8

    0.2 0

    10 20 30 40 50 60 70 80 90

    Downloads[%]

  • www.fmi.fi

    http://www.slideshare.net/tervo/ https://en.ilmatieteenlaitos.fi/open-data

of 42/42
A Smartmet Server based WFS and WCS implementation Finnish Meteorological Institute Mikko Visa, Roope Tervo
Embed Size (px)
Recommended