®
OGC
Big Data and Open Geospatial-Temporal
Information Standards
George Percivall OGC Chief Engineer
2014 Conference on Big Data from Space (BiDS '14)
Jointly organised by ESA, SatCen, JRC 12-14 November 2014, ESRIN, Frascati, Italy
© 2014 Open Geospatial Consortium
OGC ®
OGC Services Architecture
• Introduction to OGC Standards • …Service Interfaces and Encodings standards • …Grouped into semantically similar sets
OGC ®
OGC Services Architecture
WMS
WMTS WFS
Simple Features Access
Access Services
Geospatial Feature Data
Geospatial Browse/Maps
Geospatial Coverage Data
WCS
OGC ®
OGC Services Architecture
WMS
WMTS WFS
Simple Features Access
Access Services
Geospatial Feature Data
Geospatial Browse/Maps
Geospatial Coverage Data
WCS
Sensors
Puck
SOS SPS O&M SensorML
Sensor Web Enablement
Discover Task Access
OGC ®
OGC Services Architecture
Other Data
Processing Services OpenMI WPS TJS WCPS
WMS
WMTS WFS
Simple Features Access
Access Services
Geospatial Feature Data
Geospatial Browse/Maps
Geospatial Coverage Data
WCS
Sensors
Puck
SOS SPS O&M SensorML
Sensor Web Enablement
Discover Task Access
OGC ®
OGC Services Architecture
Other Data
Processing Services OpenMI WPS TJS WCPS
Geospatially Enabled Metadata
Discovery Services
CSW OpenSearch Geo ebRIM
WMS
WMTS WFS
Simple Features Access
Access Services
Geospatial Feature Data
Geospatial Browse/Maps
Geospatial Coverage Data
WCS
Sensors
Puck
SOS SPS O&M SensorML
Sensor Web Enablement
Discover Task Access
OGC ®
OGC Services Architecture
Other Data
Processing Services OpenMI WPS TJS WCPS
Geospatially Enabled Metadata
Discovery Services
CSW OpenSearch Geo ebRIM
WMS
WMTS WFS
Simple Features Access
Access Services
Geospatial Feature Data
Geospatial Browse/Maps
Geospatial Coverage Data
WCS
Other Services Workflow, Alerts
Sensors
Puck
SOS SPS O&M SensorML
Sensor Web Enablement
Discover Task Access
OGC ®
SLD
SE
Data Models and Encodings WMC
FE
GML
GeoXACML
KML
CityGML OpenGeoSMS
IndoorGML GeoSparql
WaterML GeoPackage
NetCDF GMLJP2
OGC Services Architecture
Other Data
Processing Services OpenMI WPS TJS WCPS
Geospatially Enabled Metadata
Discovery Services
CSW OpenSearch Geo ebRIM
WMS
WMTS WFS
Simple Features Access
Access Services
Geospatial Feature Data
Geospatial Browse/Maps
Geospatial Coverage Data
WCS
Other Services Workflow, Alerts
Sensors
Puck
SOS SPS O&M SensorML
Sensor Web Enablement
Discover Task Access
OGC ®
Visualization / Decision Tools and Applications GeoAPI
OpenLS
SLD
SE
Data Models and Encodings WMC
FE
GML
GeoXACML
KML
CityGML OpenGeoSMS
IndoorGML GeoSparql
WaterML GeoPackage
NetCDF GMLJP2
OGC Services Architecture
Other Data
Processing Services OpenMI WPS TJS WCPS
Geospatially Enabled Metadata
Discovery Services
CSW OpenSearch Geo ebRIM
WMS
WMTS WFS
Simple Features Access
Access Services
Geospatial Feature Data
Geospatial Browse/Maps
Geospatial Coverage Data
WCS
Other Services Workflow, Alerts
Sensors
Puck
SOS SPS O&M SensorML
Sensor Web Enablement
Discover Task Access
OGC ®
ESA HMA and OGC
• Foreword by Mark Reichardt, OGC CEO and President
– Comprehensive guide to the use of international standards to enable a European EO system of systems.
– “Indispensable resource for organisations with EO assets that wish to connect to and build a local to global system of systems”
• OGC Working Groups of major importance to HMA
– Service Architecture – Sensor Web Enablement – Catalogue – Metadata – Coverages – Compliance
© 2014 Open Geospatial Consortium 10
Thomas Usländer (Fraunhofer IOSB) Yves Coene (Spacebel s.a.) Pier Giorgio Marchetti (ESA)
OGC ®
ESA HMA Protocols under Configuration Control http://wiki.services.eoportal.org/tiki-index.php?page=HMA+Configuration+Management+Table
© 2014 Open Geospatial Consortium 11
Service Standard Status Collection Metadata OGC 11-035r1, EO Collection and Service Discovery using the ebRIM Application Profile of CSW 2.0, version
1.0, 26/03/2013. OGC Best Practice
Collection& Service Discovery
OGC 13-084, I15 (ISO19115 Metadata) Extension Package of CSW ebRIM profile, version 1.0, 28/04/2014 OGC Implementation Specification
OGC 10-032r8, OpenSearch Geo and Time Extensions, version 1.0.0, 14/04/2014. OGC Implementation Standard OGC 13-026r5, OpenSearch Extension for Earth Observation, DRAFT 1.0.0, 29/10/2014 OGC SWG Internal document Earth Observation Product Metadata
OGC 06-080r4, GML Application Schema for EO Products, Version 1.0.0, 25/02/2010 OGC Profile
OGC 10-157r3, Earth Observation Metadata Profile of Observations and Measurements,Version 1.0, 12/06/2012 OGC Implementation Standard Product Discovery OGC 06-131r6, EO Products Extension Package for ebRIM Profile of CSW 2.0, Version 1.0.0, 10/02/2010 OGC Implementation Standard OGC 10-032r8, OpenSearch Geo and Time Extensions, version 1.0.0, 14/04/2014. OGC Implementation Standard OGC 13-026r5, OpenSearch Extension for Earth Observation, DRAFT 1.0.0, 29/10/2014 OGC SWG Internal document Order OGC 06-141r6. Ordering Services Framework for Earth Observation Products Interface Standard, Version 1.0.0,
09/01/2012 OGC Implementation Specification
OGC 13-042, RESTful Encoding of Ordering Services Framework for Earth Observation Products, version 1.0, 28/04/2014.
OGC Best Practice
Feasibility Analysis OGC 10-135, Sensor Planning Service Interface Standard 2.0 Earth Observation Satellite Tasking Extension, version 2.0, 28/03/2011
OGC Implementation Standard
OGC 13-039, OpenSearch Extension for Earth Observation Satellite Tasking, OGC Best Practice OGC 14-012r1, RESTful encoding of Sensor Planning Service for Earth Observation Satellite Tasking, version
1.0.0, 17/07/2014. OGC Best Practice
Web Map Service (WMS)
OGC 07-063r1 OpenGIS WMS EO Application Profile Version 0.3.3, 05/11/2009 OGC Best Practice
Web Coverage Service (WCS)
OGC 09-110r4 WCS 2.0 Interface Standard - Core, Version 1.0.0, 10/07/2012. OGC 09-146r2 GML 3.2.1 Application Schema - Coverages (1.0.1) OGC 09-147r3 WCS 2.0 Interface Standard - KVP Protocol Binding Extension (1.0.1) OGC 09-149r1 WCS 2.0 Interface Standard - XML/SOAP Protocol Binding Extension (1.0) OGC 09-148r1 WCS 2.0 Interface Standard - XML/POST Protocol Binding Extension (1.0) OGC 09-153 WCS 2.0 Overview: Core and Extensions (Best Practice) OGC 10-164, OGC GML 3.2.1 AS – Coverages – CF NetCDF Coverage Encoding Extension, version 2.0.0 OGC 12-108, OGC GML 3.2.1 AS – Coverages – GML JPEG2000 Coverage Encoding Extension, version 0.0.4.
OGC Interface Standard
OGC 12-100r1 OGC GML 3.2.1 Application Schema – Coverages – GeoTIFF Coverage Encoding Profile, v.1.0 OGC Interface Standard OGC 10-140r1 OGC Web Coverage Service 2.0 Interface Standard - Earth Observation Application Profile
v.1.0.0, 26/02/2014. OGC Interface Standard
Web Processing Service (WPS)
No standards under configuration control
Download Service OGC 13-043, Download service for EO Products, Version 1.0, 31/01/2014. OGC Best Practice Identity Management
OGC 07-118r9, User Management Interfaces for Earth Observation Services, Version 1.1, 28/04/2014. OGC Best Practice
!
OGC ®
OGC Big Data Working Group
• Open forum on how to use OGC standards for Big Data interoperability, access, and especially analytics – Encourage collaborative development among participants
representing many organizations and communities, – Ensure appropriate liaisons to other Big Data relevant working
groups, both inside and outside OGC.
• Big Data agenda – September 2014 – GeoWave Introduction and Overview (NGA) – Well Known Text for Accumulo (NGA) – Spatio-Temporal Data Cube Analytics (Jacobs Univ.) – Elasticity for WMS Hosted in a Cloud (Amazon) – NOAA Big Data Partnership RFI (NOAA)
© 2014 Open Geospatial Consortium 12
http://www.opengeospatial.org/projects/groups/bigdatadwg
OGC ®
Discrete Global Grid Systems
Slide Source: Matt Purss, Geoscience Australia
ANZLIC National Nested Grid***
SCENZ-Grid by CSRIO for NZ
Earth System Spatial Grid by China
Snyder Grid e.g. PYXIS
*** See BIDS 2014 paper: Iterating Petabyte-scale Earth Observation processes in the Australian Geoscience Data Cube, Simon Oliver (Geoscience Australia)
OGC ®
Discrete Global Grid System (DGGS) Standards Working Group (SWG)
• Develop common criteria that will define conformant DGGSs – Considering Goodchild criteria
• Develop conceptual standard to facilitate data fusion between DGGSs using OGC Standards – to make them interoperable – with
conventional and other DGGS data – to standardize operations on them
• Engage stakeholders to encourage new use cases and adoption of interoperability through DGGSs
http://www.opengeospatial.org/projects/groups/dggsswg
OGC ®
Big data in OGC Testbeds
• OGC Interoperability Program – Prototyping unites users and industry in accelerating interoperability – 85 initiatives since 1999: Testbeds, Pilots, Experiments
• OGC Testbeds with Big Data from space – OWS-8: Observation Fusion – OWS-9: OWS Innovations – OGC Testbed 10: Open Mobility and Cloud – OGC Testbed 11: Urban Climate Resilience
• Highlights discussed in the following categories – Maps; Coverages; Performance
© 2014 Open Geospatial Consortium 15
OGC ®
Maps and Big Data
• Web Map Tiling Service (WMTS) – WMTS is WMS tuned for high velocity maps – Serves map tiles with predefined content, extent, and resolution – TileMatrixSet: collection of tile matrices defined at different scales
• OWS-9 evaluated diverse raster map tiling schemes – WMTS, Tile Map Service (OSGeo), MBTiles, TileCache
(MetaCarta), and various others
• Recommendations forwarded to OGC Standards Program. – Basis for “WMTS – Simple” based on limited TileMatrixSet – Revised WTMS standard nearing completion in SWG
© 2014 Open Geospatial Consortium 16
OWS-9 Map Tiling Methods Harmonization (Editor Joan Maso, OGC Doc 12-157)
OGC ®
WMTS Example: Several sources, Same TileMatrix
Copyright © 2013 Open Geospatial Consortium
a.tile.openstreetmap.org mt1.google.com otile1.mqcdn.com/tiles/1.0.0/map
tile.opencyclemap.org/cycle
ecn.t0.tiles.virtualearth.net
www.ogc.uab.cat/SITiled/GeoEye_Haiti_M
X=9798 Y=14664
X=9799 Y=14664
X=9800 Y=14664
X=9798 Y=14665
X=9799 Y=14665
X=9800 Y=14665
OGC ®
OWS-8 Coverages as Big Data
• WCS 2.0 Earth Observation Application Profile (EO-AP) – Build on WCS 1.0 EO-AP developed in ESA’s HMA project.
• …to include data models for: – 2-D Coverage with latitude and longitude axes, e.g., a hyperspectral satellite scene. – Subset of a 2-D satellite scene (either radar or multispectral) – Time series of co-registered 2-D satellite scenes – Stitched Mosaic as a 2-D horizontal for several co-referenced non-overlapping datasets – Dataset Series as a collection of coverages;
• …supporting these data products: – MODIS – ENVISAT MERIS L3 demonstration products (http://earth.eo.esa.int/level3/) – ENVISAT ASAR Wide Swath – Limb-scanning instruments (e.g. Microwave Limb Sounder) typically produce vertical profiles, – Narrow-swath instruments (e.g., CALIPSO) produce vertical cross-sections.
• …and these encodings: – GeoTIFF, netCDF / CF-netCDF, JPEG2000, HDF-EOS
• With a WCS 2.0 EO-AP Compliance Test and Reference Implementation
Copyright © 2011 Open Geospatial Consortium
OWS-8 WCS 2.0 EO Application Profile ER (11-096) OWS-8 WCS 2.0 EO Application Profile Compliance Tests and Reference Implementation ER (11-095) OWS-8 ER Metadata Mapping between NASA ECS/HDF-EOS and WCS 2.0 (11-090)
OGC ®
OWS-8 Coverage Processing
• WPS for EO data analysis of multiple coverages and other data
• WCPS for advanced access to EO Coverages – multi-coverage fusion
capabilities – queries taken from real-life
examples
© 2014 Open Geospatial Consortium 19
OWS-8 Geoprocessing of EO (Editor P. Baumann, OGC 11-116)
WCPS and WPS Orchestration
OGC ®
OWS-8 Big Data Analytics on Motion Imagery
© 2011 Open Geospatial Consortium, Inc.
Display to Common Operational Picture …and send to Mobile Client
References: Ingo Simonis, IGSi OGC Doc 11-108 R. Cass, M. Simms, Compusult, OGC Doc 11-134
OGC WPS for generating track features STANAG 4676 represented as GML features VMTI Implemented in OWS-8
OGC ®
VMTI to GeoSMS OWS-8 Mobile Client Implementation
• Android app for OGC GeoSMS, WFS features and KML • Mobile client receives GeoSMS with URL for WFS track feature
Copyright © 2012, Open Geospatial Consortium
Videos of OWS-8 Moving Object Tracking http://www.opengeospatial.org/pub/www/ows8/tracking.html
OGC ®
OWS-9 Access to Big Data Coverages
• OWS-9 WCS and OPeNDAP – DAP and WCS compliment each other. – Easy to use DAP to ‘reach into’ files and access their components – WCS provides specialized interface for geospatial data
• OWS-9 WCS for LIDAR – Implementation of WCS 2.0 service that serves the LIDAR data in
NITF format – service supports the JPEG 2000 output format along with GMLJP2
metadata description as well as the JPIP protocol to deliver the output JPEG2000 data.
– Demonstrated WCS with JPIP streaming for GMLJP2.
© 2014 Open Geospatial Consortium 22
OWS-9 Coverage Access ER (Editor P. Bauman, OGC 12-095) OWS-9 OWS Innovations WCS for LIDAR Engineering Report (Editor W. Han et.al., OGC 12-155)
OGC ®
OGC Testbed 9: LiDAR Geoprocessing with OGC Standards
© 2014 Open Geospatial Consortium 23
OWS-9 OWS Innovations WCS for LIDAR Engineering Report (Editor W. Han et.al., OGC 12-155)
OGC ®
Performance of OGC Services in the Cloud
• OGC service standards have been widely adopted and deployed across the world but little information has filtered out regarding the performance and scalability of products based on OGC standards.
• Easy access to large IT computing resources such as the Amazon EC2 Cloud infrastructure provides an opportunity to use a flexible and low cost IT resource environment to investigate the performance and scalability of products based on OGC standards.
• OGC Testbed 10 tested web mapping and other geo-processing use cases as a way to characterize the performance of OGC data services deployed in Cloud infrastructures.
© 2014 Open Geospatial Consortium 24
OGC ®
OGC Testbed 10: WMTS Massively Available with Cloud Services
© 2014 Open Geospatial Consortium
Concurrent user access to OGC-compliant map servers using an Amazon EC2 with an S3 network attached storage configuration
OGC ®
OGC Testbed 10: Performance & Scalability of WMS/WMTS
• Understand how OGC services perform on clouds through testing in OWS-10 Testbed
• Response time with concurrent users for – WMS: linear – WMTS: flat to 1800
!
OGC Testbed 10 Performance of OGC Services in the Cloud ER (Editor Edric Keighan, OGC 14-028r1)
OGC ®
OGC Testbed 10: SAR Interferometry with WPS on SBAS Cloud
• Performance enhancements with Cloud deployment of SBAS (Small Baseline Subset) processing application using WPS and OpenSearch OGC Web Services
© 2014 Open Geospatial Consortium 27
!
• Exploit 64 differential SAR scenes for the generation of time series showing ground displacements over a decade in geological sensitive areas.
• Part of an ongoing effort from ESA, CNR-IREA and Terradue partners
OGC ®
OGC Testbed 10: Moving Cloud Processing to Data
1. An OGC WPS Service “bought” on Terradue Cloud Marketplace
2. then deployed to Amazon UK Cloud
3. which computes using data from Interoute Cloud UK
Experiments led by Terradue in OGC Testbed 10
OGC Testbed 10 Performance of OGC Services in the Cloud ER (Editor Edric Keighan, OGC 14-028r1)
As a result of this testbed: User organizations able to request service providers (data provider, algorithm provider) to operate under a multi-tenant Cloud environment with specific performance constraints, in order to build and run scientific applications that deliver added-value products.
OGC ®
OGC Testbed 11: Urban Climate Resilience
Copyright © 2014 Open Geospatial Consortium
• simulation models integration • high-resolution data • auxiliary data integration • data representation • WFS-T Transactions
• Desktop/mobile clients • Geosynchronization • REST and GeoJSON
• What-if scenarios & models • Social/political data (TJS) • GeoPackage creation • Georeferenced grid harmon. • Streaming • Alerting
• GeoPackage creation • GeoPackage synchronization • Image streaming tests
OGC Testbed 11 UCR Thread Architect: Dr. Ingo Simonis, OGC Director Interoperability and Science
OGC ®
OGC Testbed 11 Threads RFQ/CFP: http://www.opengeospatial.org/standards/requests/126
© 2014, Open Geospatial Consortium 30
Geo4NIEM
• NIEM V3.0 IEPD Geoprocessing Round-Trip
• NIEM Enhancements Security Tagging
• NIEM-GML Feature Processing
• Security: Identification Authorization, Access
• And more…
Urban Climate Resilience (UCR)
• Climate/Big Data processing - WPS
• Hi-Res Flood Model
• Social Media
• WFS-T with REST
• GeoPackage, GeoSync
• Georeferenceable Grid Harmonization
• And more…
Cross-Community Interoperability (CCI)
• Aviation: NOTAM, AFX
• REST and SOAP
• JSON/GeoJSON
• Semantic Broker: Social Media, Linked Data, GeoSPARQL
• SPARQL for Symbology
• Compliance
• And more…
Cross Thread Scenario: Displaced Population Due to Coastal Inundation
OGC ®
Relevant OGC Engineering Reports
These engineering reports (ER) provide more detail on the topics addressed in OGC Testbeds listed above: • OWS-8 ER Metadata Mapping between NASA-EOS and WCS 2.0 (11-090) • OWS-8 WCS 2.0 EO Application Profile ER (11-096) • OWS-8 WCS 2.0 EO AP Compliance Tests and Reference Implementation ER
(11-095) • OWS-8 Geoprocessing of EO (11-116) • OWS-9 Map Tiling Methods Harmonization (12-157) • OWS-9 Data Quality for Web Mapping ER (12-160) • OWS-9 Coverage Access ER (12-095) • OGC Testbed 10 Service Integration ER (14-013) • OGC Testbed 10 Performance of OGC Services in the Cloud ER (14-028r1)
© 2014 Open Geospatial Consortium 31
OGC ®
For Details on OGC Standards…
• OGC Standards – Freely available – www.opengeospatial.org/standards
• OGC Reference Model (ORM) – Overview of OGC Standards Baseline – Resource for defining architectures for specific
applications – www.opengeospatial.org/standards/orm
George Percivall, gpercivall at opengeospatial.org