Date post: | 15-Jul-2015 |
Category: |
Technology |
Upload: | roope-tervo |
View: | 264 times |
Download: | 0 times |
Open Meteorological Data as
Part of Big Data
Finnish Meteorological Institute
Roope Tervo, Mikko Visa, Tarja Riihisaari
Finnish Meteorological Institute opened its data in 2013.
Basically everything that FMI has property rights was opened.
Data is provided in freely in machine readable format.
30.1.2015 Open Meteorological Data as part of Big Data / Roope Tervo 2
FMI Open Data
https://en.ilmatieteenlaitos.fi/open-data
Data set Description Time
Interval
Estimated
publish date
Weather
Observations
Temperature, Wind,
Humidity, Ground
Temperature…
10 min Open,
older data to be
added
Sun Radiation UV, Short and Long
Term Radiation…1 min Open
Marine
Observations
Waves, Sea
Temperature, Sea
Level…
1 h Open
Weather Radars Precipitation Rate,
Precipitation Amount…5 min Open,
older data to be
added
Lightning Thunder Strikes in
Finland5 min Open
30.1.2015 Open Meteorological Data as part of Big Data / Roope Tervo 3
Example of Data Sets
30.1.2015 Open Meteorological Data as part of Big Data / Roope Tervo 4
Example of Data Sets
Data set Description Time Interval Estimated
publish date
Real Time
Observations
Real Time Observations from
specific location(s)
AWS 2010 –
Soundings 1959 –
Flashes 1998 –
Sea Level 1971 –
Waves 2005 –
Open
older data will
be added
Climatological
Observations
Dayly and monthly
temperature mean and
extreme values from weather
stations
1959 - Open
Climatological
Observations
Monthly temperature and
precipitation rate mean
values interpolated to grid
1961 - Open
Climatological
Reference
Climatological Reference.
Temperature, humidity,
pressure, precipitation
amount and snow depth.
Reference seasons:
1971-2000 1981-
2010
Open
30.1.2015 Open Meteorological Data as part of Big Data / Roope Tervo 5
Example of Data Sets
Data set Description Time Interval Estimated
publish date
Weather forecast
model HIRLAM RCR
Point forecasts and grid
dataLatest model
run
(4 times a day)
0…54 h
Open
Sea forecast models Sea level point
forecasts, Wave (WAM)
and current (HBM) as
grid data
Latest model
run
(4 times a day)
0...54 h
Open
Environmental
Monitoring Facilities
Weather observation
stations, radars…2015
Aviation
Observations
METAR 30 min open
Ground & mast
observations
Special observations
from ground and masts2015
30.1.2015 Open Meteorological Data as part of Big Data / Roope Tervo 6
Example of Data Sets
Data set Description Time Interval Estimated
publish date
Air Quality
Observations
Air Quality Observations 1h 2015-2016
Silam Model Dispersion Model for Air
Quality, Forest Fire and
Pollen
Latest model
run (once a day)
0…96h
2015
HELMI Ice Model Ice forecast model Latest model
run
(4 times a day)
0...54 h
open
Soundings Temperature, Humidity,
Pressure, Wind from
ground to 25 km height
2 times a day 2015
FMI Open Data Portal follows INSPIRE requirements.
30.1.2015 Open Meteorological Data as part of Big Data / Roope Tervo 7
FMI Open Data Portal
Meta data
Data Models
Services
The very same data portal works as Open Data and
INSPIRE portal.
Catalog Service
(CSW)
o Based on GeoNetwork
30.1.2015 8Open Meteorological Data as part of Big Data / Roope Tervo
View Service (WMS)
o Based on GeoServer
o Only the most common layers
published
Download Service
(WFS 2.0)
o Web Feature Service (WFS) 2.0 Simple Profile
o Based on stored queries
o Predefined data sets with possibility for additional parameters (i.e. time and area)
o In-house production
30.1.2015 9Open Meteorological Data as part of Big Data / Roope Tervo
Data Models
o Observations and point forecasts as GML
o The same data is published in two different formats.
o Gridded data is provided in
appropriate binary format (grib,
NetCDF, HDF…)
o WFS members contains the metadata ‘envelope’ with a link to a actual data
o Every XML document as self-
explanatory as possible
30.1.2015 10Open Meteorological Data as part of Big Data / Roope Tervo
Weather Data as
Part of Big Data
• Important constituent
• Weather impacts to everything and each one of us.
• New technology makes us even more fragile of weather.
30.1.2015 11Open Meteorological Data as part of Big Data / Roope Tervo
Traffic
Advertising
Energy
Retail trade
Outdoor
activities
Weather data – is it big?
30.1.2015 12Open Meteorological Data as part of Big Data / Roope Tervo
Snijders, C.; Matzat, U.; Reips, U.-D. (2012). "'Big Data': Big gaps of knowledge in the field of Internet". International Journal of Internet Science 7:
1–5
Weather data – is it big?
30.1.2015 13Open Meteorological Data as part of Big Data / Roope Tervo
“Data sets with sizes beyond the ability of commonly used software tools to capture,
curate, manage, and process data within a tolerable elapsed time.“Snijders, C.; Matzat, U.; Reips, U.-D. (2012). "'Big Data': Big gaps of knowledge in the field of Internet". International Journal of Internet Science 7:
1–5
Weather data – is it big?
30.1.2015 14Open Meteorological Data as part of Big Data / Roope Tervo
• In-situ weather measurement 1 TB
• Weather radar data 50 TB
• NWP model gridded data (FMI)
• HIRLAM 120 TB
• AROME 230 TB
• Satellite image data (FMI)
• Globsnow 32 TB
• Other 50 TB
• Climate model data (FMI) 29 TB
• Other models
• SILAM 100 TB
• Tuuliatlas 21 TB
• Other 300 TB
FMI Open Data as
Part of Big Data
• Tried to tackle big data challenges
• Machine readable
• Follows standards
• Rich meta data• Every XML document as self-
explanatory as possible
• Users don’t need to download everything, they can fetch only the parts of their interest
• parameters, area, time, etc…
• Quality information included (where applicable)
30.1.2015 15Open Meteorological Data as part of Big Data / Roope Tervo
30.1.2015 Open Meteorological Data as part of Big Data / Roope Tervo 16
MetoLib
o Open source JavaScript library produced by Finnish Meteorological Institute
o Helps users to load and use the data
o Python version is on the list
Easyrequests Cache
Parse the data to as JSON
30.1.2015 Open Meteorological Data as part of Big Data / Roope Tervo 17
Data Contents
Popularity Comparison
91
5 40
102030405060708090
100
Radar Images(WMS + WFS)
Forecast (WFS) WeatherObservations
(WFS)
Downloads[%]
o Based on year 2014
o All data sets combined
o All radar image queries treated as data queries
o Note that some data is updated more often than others
And a little over300 000 data downloads
per day(3,7 req/s)
At the moment about 7200
registered users
30.1.2015 20Open Meteorological Data as part of Big Data / Roope Tervo
Some Experiences
Practicallyno clientsupportscomplexfeatures
Although standards are followed, there’s
a gap between provided data model
and clients’ capabilities
30.1.2015 21Open Meteorological Data as part of Big Data / Roope Tervo
Some Experiences
GeoServer is modified to support
stored queries in WFS 2.0 (will be
integrated into the standard package
Q1/2015)
FMI is going to open the same data as simple features to
support clients
30.1.2015 22Open Meteorological Data as part of Big Data / Roope Tervo
Some Experiences
Industry ishappy to usestandardized
services
Amateur andfreelancer coders
would prefer simple JSON API
30.1.2015 23Open Meteorological Data as part of Big Data / Roope Tervo
Some Experiences
So far maybe
even more
professional
interest than
private
Quite many
expected a user
interface to load data
to i.e. to Excel
instead of machine
readable interface
30.1.2015 24Open Meteorological Data as part of Big Data / Roope Tervo
Some Experiences
For now,
very few have
been interested in
forecast models
as a grid data
Point forecasts,
observations and
radar images are the
most interesting data
types
30.1.2015 25
Some Experiences
Open Meteorological Data as part of Big Data / Roope Tervo
Finally we have
all the data behind
one access point
behind standard
interfaces and
in harmonized
formats
The Open Data
project has been a
great boost for
ourselves both
technically and for
working culture
30.1.2015 26
Some Experiences
Open Meteorological Data as part of Big Data / Roope Tervo
www.fmi.fi
http://www.slideshare.net/tervo/
https://en.ilmatieteenlaitos.fi/open-data