A Smartmet Server based WFS and WCS implementation
Finnish Meteorological Institute
Mikko Visa, Roope Tervo
Finnish Meteorological Institute opened its data in 2013. Basically everything that FMI has property rights was opened. Data is provided freely in machine readable format.
23/09/16 2
FMI Open Data
https://en.ilmatieteenlaitos.fi/open-data
Introduction
23/09/16 A Smartmet Server based WCS implementation
Roope Tervo, Mikko Visa
3
• WFS and “WCS-like” download service implementations in production • WCS implementation currently work in progress
23/09/16
Serving Coverage Data in FMI Open Data Portal Roope Tervo, Mikko Visa
4
Implementations at FMI
WFS implementation
23/09/16 Serving Coverage Data in FMI Open Data Portal
Roope Tervo, Mikko Visa
5
• Web Feature Service (WFS) 2.0 Simple Profile • Based on stored queries
• Pre-defined data sets with possibility for additional parameters (ie. time and area)
• In-house production, based on Smartmet Server • Currently being opensourced (https://github.com/fmidev)
• In production use since 04/2013 • Observations and point forecasts in GML format
• Grid Series Observation/Point Time Series Observation/Simple Feature
• Gridded data in appropriate format • WFS members contain the metadata ‘envelope’ with a link to the actual data • GRIB1/2 for weather models, NetCDF for sea models, GeoTIFF for radar images
23/09/16
Serving Coverage Data in FMI Open Data Portal Roope Tervo, Mikko Visa
6
WFS download service
Producing INSPIRE Data Products Point Forecasts
23/09/16 7
File System
Smartmet Server
Memory mapped data.
Server provides logic for
interpolating the data for
requested area and time.
Smartmet Server WFS
Plugin
Producing INSPIRE Data Products Grid Forecasts 1/2
File System
Smartmet Server
Server provides relevant meta
data to fetch the data content
Smartmet Server WFS
Plugin
23/09/16 8
Producing INSPIRE Data Products Grid Forecasts 2/2
File System
Smartmet Server
Smartmet Server
Download Plugin
Memory mapped data.
Server provides logic for
interpolating the data for
requested area and time.
23/09/16 9
• Provides grid data as binary data • Supported output formats: GRIB1, GRIB2, NetCDF and Querydata • Supports all proj.4 projections (depends on output format support) • Supports slicing by
• area (bbox)
• elevation (pressure and/or model level)
• time (start time, end time and origin time) • Possibility to define grid resolution by
• selecting every Nth grid point to x and y direction
• grid size à data is interpolated to new grid points 23/09/16
Serving Coverage Data in FMI Open Data Portal Roope Tervo, Mikko Visa
10
WCS-like download service
GetCapabilities: • Describes capabilities of the service • Normally start point for client applications • http://data.fmi.fi/fmi-apikey/fmi-apikey/…/wfs?
request=GetCapabilities
23/09/16 FMI Open Meteorological Data Services | Roope Tervo 11
WFS Basics observations and point forecasts
DescribeStoredQueries • List and describes available stored queries (pre-defined
data sets) • Stored query name and description
• Possible additional parameters available for the stored query
• http://en.ilmatieteenlaitos.fi/open-data-manual-fmi-wfs-services
23/09/16 FMI Open Meteorological Data Services | Roope Tervo 12
WFS Basics observations and point forecasts
GetFeature • Actual data query • In FMI service, stored query id is mandatory
• &storedquery_id=..
• Other possible parameters depend on the stored query • Possible values can be fetched with DescribeStoredQueries
request.
• Location has to be often defined somehow.
http://data.fmi.fi/fmi-apikey/.../wfs?request=GetFeature&storedquery_id=fmi::observations::weather::cities::timevaluepair&bbox=21,62,22,64
23/09/16 FMI Open Meteorological Data Services | Roope Tervo 13
WFS Basics observations and point forecasts
Location parameters in stored queries & bbox à area
• By default, in same CRS with the (requested) CRS. Specify as fifth element if needed (i.e. &bbox=132249,6433579,631999,6933329,epsg:3067)
& fmisid à FMI id for observation station & wmo à WMO id for observation station
23/09/16 FMI Open Meteorological Data Services | Roope Tervo 14
WFS Basics observations and point forecasts
Location parameters in stored queries & geoid à http://geonames.org id for location & place à human readable name
• use region to accurate your query (i.e. &place=kumpula,tampere)
• can be used several times in one query (i.e. &place=kumpula,helsinki&place=heinola)
& latlon à coordinates (used only in marine data sets)
23/09/16 FMI Open Meteorological Data Services | Roope Tervo 15
WFS Basics observations and point forecasts
GetPropertyValue • Responses only requested part of the GetFeature response • Use ValueReference=xpath to define part of interest • Same additional parameters for stored query with GetFeature
request are still valid • For example:
http://data.fmi.fi/fmi-apikey/…/wfs?request=GetPropertyValue&storedquery_id=fmi::observations::weather::multipointcoverage&place=kumpula,tampere&ValueReference=wfs:FeatureCollection/wfs:member/omso:GridSeriesObservation/om:result
23/09/16 FMI Open Meteorological Data Services | Roope
Tervo 16
WFS Basics observations and point forecasts
• WFS response contains the same meta data information as in observations and point forecasts
• om:result contains gmlcov:RectifiedGridCoverage • Basically the same with Grid Series Observation
• domainSet defines the grid (now as regular grid)
• rangeSet contains data as an external link to the binary content • rangeType defines the parameters
23/09/16 FMI Open Meteorological Data Services | Roope Tervo 17
WFS basics binary data
• Note that fileReference points often to an other service • Can also be used directly, but
• It is always good practice to consult WFS for available times and parameters
23/09/16 FMI Open Meteorological Data Services | Roope Tervo 18
WFS basics binary data
Client
WFS
Geoserver WMS
Smartmet Server
Available radar images? Available weather model outputs?
• Area, time and (weather) parameters may be defined in the request
• For weather models, intersection of requested area and available data area is returned
• For radar images, all images which intersects requested spatial and temporal space are returned
23/09/16 FMI Open Meteorological Data Services | Roope Tervo 19
WFS basics binary data
Requested area
Radar images
Radar images Requested area
Model coverage
Weather models
Radar images • The reference points to a “original” gray scale GeoTiff image • Images can also be downloaded as color images
• Remove &styles=raster from the data request
• But then information is lost
• Used SLD-files can be downloaded from https://github.com/fmidev/opendata-resources/tree/master/sld
• Consult om:parameter element for single radar measurement parameters
• Scanning angle, bin count and bin length
23/09/16 FMI Open Meteorological Data Services | Roope Tervo 20
WFS basics binary data
Numerical models • The reference points to (often a subset) the model output in
appropriate format • GRIB1/2 for weather models
• NetCDF for marine models
• Note that whole model output can be large • Up to 17 GB
• Tip: download always only area, parameters and time range of interest
23/09/16 FMI Open Meteorological Data Services | Roope Tervo 21
WFS basics binary data
WCS implementation (in progress)
23/09/16 Serving Coverage Data in FMI Open Data Portal
Roope Tervo, Mikko Visa
22
• Well proven existing data server (Smartmet Server) • Currently being opensourced! (https://github.com/fmidev)
• WFS implementation already based on Smartmet Server in production since 04/2013 • Why not use already existing servers
• Geoserver WCS only returns multipart format,GRIB/NetCDF support missing • More difficult integration to FMI’s own production system
23/09/16
Serving Coverage Data in FMI Open Data Portal Roope Tervo, Mikko Visa
23
Why build or own WCS implementation?
• Version 2.0.0 and 2.0.1 • GetCapabilities
• INSPIRE metadata included • DescribeCoverage
• validtime, area, level • GetCoverage
• Whole data in native CRS (RectifiedGridCoverage) or • outputCrs=http://www.opengis.net/def/crs/EPSG/0/XXXX
• Slice or trim: subset (x,y,z,t) using KVP/GET or XML/POST • Time can be given in ISO or unixtime format
• Formats: NetCDF partially implemented, GRIB1/2 coming later • format=application/netcdf
23/09/16
Serving Coverage Data in FMI Open Data Portal Roope Tervo, Mikko Visa
24
Supports
• HIRLAM model temperature trimmed to selected area, output in NetCDF / WGS84
• http://beta.fmi.fi/wcs?service=WCS&request=GetCoverage&coverageid=hirlam-temperature&format=application/netcdf&subset=x(-30,30)&subset=y(60,90)&outputCrs=http://www.opengis.net/def/crs/EPSG/0/4326
23/09/16
Serving Coverage Data in FMI Open Data Portal Roope Tervo, Mikko Visa
25
Examples (WCS)
• HBM salinity, one level, one timestep
• http://beta.fmi.fi/wcs?service=WCS&request=getCoverage&coverageid=hbm-salinity&output=application/netcdf&subset=z(400)&subset=t(“2016-09-15T13:00:00Z”)
• HBM salinity, one level, time interval • http://beta.fmi.fi/wcs?service=WCS&request=getCoverage&coverageid=hbm-salinity&format=application/netcdf&subset=z(400)&subset=t(“2016-09-16T13:00:00Z”,”2016-09-16T14:00:00Z”)
23/09/16
Serving Coverage Data in FMI Open Data Portal Roope Tervo, Mikko Visa
26
Examples (WCS)
• HBM salinity, one level, 2 timesteps, single point, • http://beta.fmi.fi/wcs?service=WCS&request=getCoverage&coverageid=hbm-salinity&format=application/netcdf&subset=x(24.937592)&subset=y(59.858954)&subset=z(10)&subset=t(“2016-09-16T13:00:00Z”,”2016-09-16T14:00:00Z”)
• [[email protected] ~]$ ncdump /tmp/x.nc • netcdf x { • dimensions: • time = 2 ; • level = 1 ; • yc = 1 ; • xc = 1 ;
23/09/16
Serving Coverage Data in FMI Open Data Portal Roope Tervo, Mikko Visa
27
Examples (WCS)
• No support for MetOcean WCS 2.0 Profile yet • MetOcean WCS 2.0 Profile status?
• WCS 2.1 (including CIS 1.1) • GRIB output
23/09/16
Serving Coverage Data in FMI Open Data Portal Roope Tervo, Mikko Visa
28
Still to implement (WCS)
• Internal use: • Replace the “wcs-like” download plugin
• External use • Add to Open Data / INSPIRE interface as replacement for current non-standard “wcs-like” download plugin
23/09/16
Serving Coverage Data in FMI Open Data Portal Roope Tervo, Mikko Visa
29
Usage planned for WCS
Implementation challenges
23/09/16 Serving Coverage Data in FMI Open Data Portal
Roope Tervo, Mikko Visa
30
• CRS/SRS challenges • many weather models are calculated in rotated lat-lon • Lat/lon order dependent on CRS/SRS, confusing..!
• How to define time interval: starttime,endtime or starttime/endtime? • How to define a dataset?
• In INSPIRE context often a new version, but this is not feasible in very frequently updated data such as weather data
23/09/16
Serving Coverage Data in FMI Open Data Portal Roope Tervo, Mikko Visa
31
Challenges 1/2
• Reprojection not well supported in WCS, however there is often need for this in the MetOcean domain • Lack of WCS 2.x clients, hard to test!
• Clients supporting 1.x exist (QGIS for example) • You have the web browser of course..
• Dimensions • Now x, y, z, t… • Currently only latest model run is offered, one solution would be to put also this in the CoverageId • How to handle different (meteorological) parameters in model data? Currently need to be different coverages..
• What about EPS, probability forecasts…
23/09/16
Serving Coverage Data in FMI Open Data Portal Roope Tervo, Mikko Visa
32
Challenges 2/2
Experiences
23/09/16 Serving Coverage Data in FMI Open Data Portal
Roope Tervo, Mikko Visa
33
• Data is often 5 dimensional with irregular grid and
time intervals
Multidimensional data is hard to
handle
Hard Parts
23/09/16 34
Names, units, levels, time intervals… Handling parameters
is one of the hardest parts.
Hard Parts
23/09/16 35
Converting everything to one
data format is complicated, time consuming and
expensive.
Multiple data formats are required
Hard Parts
23/09/16 36
And a little over 440 000 data downloads
per day (5,2 req/s)
At the moment about 10000
registered users
Some Experiences
23/09/16 37
INSPIRE is a long project. Better to look forward than
backward
Although standards are followed, there’s
a gap between provided data model
and clients’ capabilities
Some Experiences
23/09/16 38
There’s not always support for HTTP
1.1. (in clients, proxies, load balancers…)
Data need to be transferred in
chunked encoding (requires HTTP 1.1)
Hard Parts
23/09/16 39
For example many weather models are calculated in rotated
lat-lon. Supporting several projections for the
data is vital.
Lessons learned
23/09/16 40
23/09/16 41
Data Models Popularity Comparison
80
19.8
0.2 0
10 20 30 40 50 60 70 80 90
Downloads[%]
www.fmi.fi
http://www.slideshare.net/tervo/ https://en.ilmatieteenlaitos.fi/open-data