www.seegrid.csiro.au
SEEGRID Roadshow Technical Workshop
Stuart Girvan – Geoscience Australia
Rob Atkinson – Social Change Online
CSIRO - Andy Dent, Simon Cox, Robert Woodcock
SEEGrid Roadshow 2005
Disclaimer
Participants should only act on the information and advice in these presentations after independent consideration of their circumstances
SEEGrid Roadshow 2005
Program
Demonstrator Architecture
OGC's WFS Specification
Introduction to Geoserver
GML Application Schemas
Implementing Geoserver - Database and Data Mapping Issues
Use Cases and Service Structure for Clients in an Open Ended Architecture
Further Technical Issues and Discussion
SEEGrid Roadshow 2005
Contributors
Rob Atkinson, Peter Barrs – Social Change Online
Joan Esterle (UQ), Simon Cox, Rob Woodcock, Andy Dent – CSIRO EM
Greg Jenkins, Mark Jolly, Terry Hannant – PIRSA
Brendon Ward, Stephen Bandy – DOIR (GSWA)
Lesley Wyborn, Aaron Sedgmen, Tim Mackey – GA
Stuart Girvan – GA, pmd*CRC
SEEGrid Roadshow 2005
Did everyone see the demonstrator this morning?
http://cgsrv3.arrc.csiro.au/seegrid/savedapps/filter
http://www.ga.gov.au/wfs/reader/wfsGeochemReaderHome.jsp
Demonstrator
PIRSAWeb Feature Service (WFS)
Common Interface Binding – GML/XMML
GA Geochemistry
Feature Data Source
DOIRGeochemistry
FeatureData Source
DOIRWeb Feature Service (WFS)
GAWeb Feature Service (WFS)
Geoserver (Open Source)
PostGIS (Open Source)
OraclePostGIS (Open Source)
CLIENT APPLICATIONS
DATA ACCESS SERVICES
DATA SOURCES
WebMap Composer
GA Reports Application
PIRSA Geochemistry
FeatureData Source
Little or no change required here
Translation to standards here
Common Interface Binding – GML/XMML
WebMap Composer
GA Reports Application
PIRSAWFS
DOIRWFS
GAWFS
NTGSWFS
MRTWFS
NRMWFS
NSWDPIWFS
VICDPIWFS
??
?
CLIENTS
DATA SOURCES
DATA SERVICES
SEEGrid Roadshow 2005
Services
Registry
ClientApplications
publish
bind
find
Data Access Services
Simple Web Services Architecture
SEEGrid Roadshow 2005
ClientApplications
bind
• For the moment not using registries to find or publish services
• Using “Magic Knowledge”, ie applications will know the data services available
GA WFS
GSWA WFS
PIRSA WFS
Services
Simple Web Services Architecture
SEEGrid Roadshow 2005
Web Feature Service (WFS)
Specification https://portal.opengeospatial.org/files/?artifact_id=8339 (only 131 pages!)
Uses http
Accepts requests as either key-value pairs or XML
Responds in XML (GML)
SEEGrid Roadshow 2005
Web Feature Service (WFS)
Uses request/response not SOAP (although it can be wrapped in SOAP)
Three main calls:
GetCapabilities
DescribeFeatureType
GetFeature
SEEGrid Roadshow 2005
Web Feature Service (WFS)
WebFeatureService
Get Capabilities Request
Get Capabilities Response
Describe Feature Type RequestDescribe Feature Type Response
Get Feature RequestGet Feature Response
http protocol
XML/KVP
XML
XML/KVP
GMLSchema
XML/KVP
GML
Data Source
ConfigFiles
Application (web based or desktop)
SEEGrid Roadshow 2005
Features
Features ≈ geographic features
Features are Geography Markup Language (GML) encoded
GML provides a standard for representing geometries (points, lines, polygons) and properties of geographic features
SEEGrid Roadshow 2005
Features
Borehole collar location shape collar diameter length operator logs related observations …
Fault shape surface trace displacement age … Ore-body
commodity deposit type host formation shape resource estimate …
Observation location subject/specimen/station property/theme/measurand method operator date/time result (+ type/reference
system/scale/classification) …
Basin? formations shape – time
dependent resource estimate …
SEEGrid Roadshow 2005
<abc:Bridge gml:id = “b1”> <abc:span>50</abc:span> <abc:heightAtCenterSpan>51</abc:heightAtCenterSpan> <abc:constructed>10-10-1996</abc:constructed> <gml:location> <gml:Point srsName = “#myRefSys”> <gml:coordinates>12435.2, 6756.34</gml:coordinates> </gml:Point> </gml:location></abc:Bridge>
Example: Instance of bridge feature encoded as GML
GML
attributes
Features
SEEGrid Roadshow 2005
WFS Demo
GetCapabilities
DescribeFeatureType
GetFeature
SEEGrid Roadshow 2005
Filter Encoding Specification
Defines how to write WFS queries when making a GetFeature call
http://portal.opengeospatial.org/files/?artifact_id=8340
SEEGrid Roadshow 2005
Filter Encoding Example
<Filter>
<And>
<PropertyIsLessThan>
<PropertyName>DEPTH</PropertyName>
<Literal>30</Literal>
</PropertyIsLessThan>
<BBOX>
<PropertyName>Geometry</PropertyName>
<gml:Envelope srsName="http://www.opengis.net/gml/srs/epsg.xml#63266405">
<gml:lowerCorner>13.0983 31.5899</gml:lowerCorner>
<gml:upperCorner>35.5472 42.8143</gml:upperCorner>
</gml:Envelope>
</BBOX>
</And>
</Filter>
SEEGrid Roadshow 2005
How would you use a WFS?
Using http calls from your programming language to access the WFS and native parsers
Using http calls and style sheets to transform the XML
SEEGrid Roadshow 2005
Eg GA Report Pages
Java reads list of input URLs
Subs in search values to pre-prepared XML chunks
Calls URL of WFS(s) with XML parameters
Parses XML document returned by WFS
Data from XML document held in Java objects and sent to jsp pages for display
SEEGrid Roadshow 2005
Why Geoserver?
It’s the reference standard for implementation
Open Source – we could influence and directly upgrade if necessary
Open Source – no license fees/purchase cost
SEEGrid Roadshow 2005
Other WFS Products
• Deegree (open source), Cadcorp SIS Map Server, CARIS Spatial Fusion Enterprise, RedSpider Web 3.0, Map Manager, GenaWare, SclFeatureServer, JCarnacGIS, GeoMedia WebMap, MapXtreme (MapInfo), MapServer(UMN) 4.2 + more
* Source: OGC Registered Products
SEEGrid Roadshow 2005
Geoserver
Java based, servlets
Runs using Tomcat (or embedded Jetty)
Open source community size?
Reliant on and closely connected to Geotools
Lots of modules
SEEGrid Roadshow 2005
Setting up Geoserver
Install into Tomcat using WAR file or use inbuilt Jetty servlet container
Works with PostGIS, Oracle, ARCSDE, Shapefiles, mysql as data sources
Config files catalog.xml, info.xml, schema.xml directory structure
SEEGrid Roadshow 2005
Community/Application Schemas
Community schemas provide the common or shared model
The target community schema is essential
SEEGrid Roadshow 2005
GML Community Schemas
GeologicalXMML - Exploration data (Example Geochemistry)
ADX – Assay data
GPML – plate tectonics
XMML/GeoTime – stratigraphy
CGI International Model
OtherLandGML
MarineXML
SEEGrid Roadshow 2005
GML Application Schemas
Design methodology can be learnt to develop community specific GML application schemas (Simon Cox, Andy Dent CSIRO)
Equally important is who will be the custodian and manager of a community schema
Also see GML 3.1 specification , Chapter 8 Rules for Application Schemes
SEEGrid Roadshow 2005
Informational Requirements
Information models impact on real world performance
Ontology
Registries
SEEGrid Roadshow 2005
Why XML?
Extensibility
Self describing
Ability to be (remotely) validated against schema
XML Schema provides “loose tolerances”
All software languages have tools to deal with XML
SEEGrid Roadshow 2005
Traditional GIS and Standard WFS
The WFS spec states the minimum requirements for a service interface
It says nothing about private schemas vs public (community) schemas
Hence most WFS software will only produce “flat” GML
SEEGrid Roadshow 2005
Flat GML Example
Sample ID Coordinates Sample Type SiO2% TiO2% Al2O3% Fe2O3% MgO%
11111 127, -32 Whole Rock 50 1 14 14 3
<gml:featureMember> <GeochemSpecimen gmlid=”1111”>
<gml:Point srsName="http://www.opengis.net/gml/srs/epsg.xml#8307"> <gml:coordinates decimal="." cs="," ts="">118.78531,-21.122948</gml:coordinates> </gml:Point> <SampleType>whole-rock</ SampleType > <SiO2%>50</SiO2%> <TiO2%>1</TiO2%> <Al2O3%>14</Al2O3%> <Fe2O3%>14</Fe2O3%> <MgO%>3</MgO% >
</GeochemSpecimen></gml:featureMember>
SEEGrid Roadshow 2005
Flat GML Example
<SiO2%>50</SiO2%>
<SiO2%ProcedureID>1</SiO2%ProcedureID>
<SiO2%Name> Proc1</SiO2%Name>
<SiO2%Description> Blah 1</SiO2%Description>
<SiO2%Method> X1</SiO2%Method>
<SiO2%DetectionLimit>0.01</SiO2%DetectionLimit>
<TiO2%>1</TiO2%>
< TiO2%ProcedureID>2</ TiO2%ProcedureID>
< TiO2%Name> Proc2</ TiO2%Name>
< TiO2%Description> Blah 2</ TiO2%Description>
< TiO2%Method> X2</ TiO2%Method>
< TiO2%DetectionLimit>0.01</ TiO2%DetectionLimit>
Nested XML Example
<GeochemMeasurement>
<resultMeasureExactoriginal UOM="wt%">50</resultMeasureExact>
<Analyte>SiO2</Analyte>
<procedureUsed>
<AssayProcedure gml:id="1">
<name>Proc1</name>
<description>Blah 1</description>
<Method>X1</Method>
<analyteDetails>
<AnalyteSensitivity>
<Analyte> SiO2</Analyte>
<lowerDetectionLimit uom="wt%">0.01</lowerDetectionLimit>
</AnalyteSensitivity>
</analyteDetails>
</AssayProcedure>
</procedureUsed>
</GeochemMeasurement>
<GeochemMeasurement>
<resultMeasureExactoriginal UOM="wt%">1</resultMeasureExact>
<Analyte>TiO2</Analyte>
<procedureUsed>
<AssayProcedure gml:id="2">
<name>Proc2</name>
<description>Blah 2</description>
<Method>X2</Method>
<analyteDetails>
<AnalyteSensitivity>
<Analyte> TiO2</Analyte>
<lowerDetectionLimit uom="wt%">0.01</lowerDetectionLimit>
</AnalyteSensitivity>
</analyteDetails>
</AssayProcedure>
</procedureUsed>
</GeochemMeasurement>
SEEGrid Roadshow 2005
XML Structures
There are potentially many structures or information views that may be appropriate for describing data
These may be influenced by use cases and architectural considerations
SEEGrid Roadshow 2005
Data source to community schemas
Community schemas provide the common or shared model
All data providers have their own local data model
All data providers must map data from local source (database) to community schema, irrespective of technology implementation
SEEGrid Roadshow 2005
Requirement
We need to be able to map from private schemas to public in as flexible a manner as possible
SEEGrid Roadshow 2005
WFS Spec Problem
WFS spec allows any query based on the feature description (community schema)
Comes from the point of view that you’d be able to use, for example, xpath against a ‘notional’ XML document
In reality almost all data sources will be databases or proprietary formats of some kind
In practise want to limit the potential queries that can be made against the data source
SEEGrid Roadshow 2005
Query Profiles
Why you should only allow certain queries
efficiency
security
user friendliness
legal requirements
Eg phone books, amazon.com
SEEGrid Roadshow 2005
Requirement
Limit the possible queries that can be made against the feature data source
SEEGrid Roadshow 2005
Implementation
At beginning of project Geoserver (nor any other software we were aware of) could not perform mapping from private database to community schema
Geoserver could only serve “flat” GML
No restrictions on queries (follows the WFS spec)
SEEGrid Roadshow 2005
Geoserver Enhancements
Initial stages of project included upgrading Geoserver to include plug in mapping to complex schemas and ability to configure pre-defined queries
Info.xml and Schema.xml files
Catch – only works from database source that you can run an SQL query against
SEEGrid Roadshow 2005
Mapping & Query Limitations
Limited amount of branching
Restricted in the positioning of branching
No data dependent tags
Only one predefined query per feature type
SEEGrid Roadshow 2005
Coping with Limitations
The impacts of the limitations of Geoserver and WFS were accommodated by
agreeing to slightly modified target schemas
agreeing to commonly defined input parameters and patterns
a small number of technical hacks
SEEGrid Roadshow 2005
Geoserver Development
The Geotools community is considering a comprehensive solution to the problems of complex objects.
See http://docs.codehaus.org/display/GEOTOOLS/Community+Schema+Support+and+Complex+Types
If you’re very keen get involved http://geoserver.sourceforge.net/html/index.php
SEEGrid Roadshow 2005
Technical Lessons
https://www.seegrid.csiro.au/twiki/pub/Infosrvices/MCAProjectOutputs/TechnicalLearningfromtheMCA.pdf
SEEGrid Roadshow 2005
Baseline Software Requirements*
Compliant with current WFS specification
Separation of back end system from common interface
Ability to restrict incoming queries
*For equivalent of current project
SEEGrid Roadshow 2005
Baseline Software Requirements
Ability to access back end data source of choice (potentially through firewall)
Ability to produce GML documents with at least single branching complexity, element attributes, and nestable single elements
SEEGrid Roadshow 2005
Software Wishlist
Ability to map to a GML document of any complexity and a GUI interface to configure the mapping
Ability to run various queries against a single defined feature type
Ability to map to an element based on a data source value
SEEGrid Roadshow 2005
WFS Spec Changes
As a result of the project there is a formal change request in with OGC to allow for query profiles for a feature
This will also require a minor change in the Filter Encoding Specification
SEEGrid Roadshow 2005
Moving forward
•The SEEGRID Roadshow will extend the test bed to the remaining state and territory surveys for Geochemistry
•Considering a 3 year ICIP (Industry Co-operative Innovation Program) grant which will take it from a test bed to a production system (network)
The Industry in ICIP is critical
SEEGrid Roadshow 2005
Moving forward
•The establishment of a production level network of services for one data type will greatly smooth the way for other types of info, not just Geological
•Opportunities to cooperate with SEEGRID to progress technological and standards solutions
As part of ICIP proposal or independently
•SEEGRID is free, contributions to common goals come from sub sets of the community
User wants to select sample data by some bounding box
Sample data must be queryable by bounding box
Scenario 1
User wants to see details of data selected in bounding box
Sample data must be queryable by bounding box or by id numbers of selected records
Scenario 2
User wants to select sample data by some bounding box and some threshold value
Sample data must be queryable by bounding box and threshold value property
Scenario 3
SEEGrid Roadshow 2005
Client Requirements
Availability, SLA’s
Support
Response times
Data quality, metadata
SEEGrid Roadshow 2005
Server Considerations
Not all use cases/query profiles can be known
But you can probably make a reasonable guess as to what may be required
Most importantly, like any project, need to involve clients (if possible)
SEEGrid Roadshow 2005
Client Strategies for Using WFS
Querying everything vs multiple queries
Schema dependency
Caching vs going back to the web service
SEEGrid Roadshow 2005
Technical Lessons
https://www.seegrid.csiro.au/twiki/pub/Infosrvices/
MCAProjectOutputs/TechnicalLearningfromtheMCA.pdf
SEEGrid Roadshow 2005
WFS in the Notional Architecture
Notional Architecture is the end game perfect world
A brief note – what are registries about?
Imagine you’re an application!
Features Coverages ModelsReal-time
data
Data Access Services
Ontology
Registry
ServiceMetadata
ClientApplications
PersistentBindings
FeatureType
Catalog
ModelManagement
Services
publish
bind
find
DatasetMetadata
ProcessingServices
Notional Architecture
Notional architecture
Dictionary ServiceMetadata
PersistentBindings
FeatureType
Catalog
DatasetMetadata
Yellow Pages + Semantics
MANAGED
Features Coverages ModelsReal-time
data
Data Access Services
Registry
ClientApplications Model
ManagementServices
publish
bind
find
ProcessingServices
Managed Dataset MetadataSomeone has vouched that from this service you can
get a sparkplug that has a certain pedigree/qualityMetadata - it was made in Germany by Mercedes Benz
Managed Persistent BindingsSomeone has created instructions to use this service
in the following waysPersistent Binding (ie always true) - only ever put the sparkplug the right way up
For a Commodore you need sparkplug X
Managed Service MetadataSomeone has vouched that a spare parts dealer service
exists that supplies sparkplugs Service Metadata – Add in the yellow pages for the spare parts dealer
Including details of how to contact (use) that service
Managed Feature Type CatalogueFeature Type Catalogue – List of Feature Types eg Spark Plugs,
Petrol Caps, Oil FiltersFeature Type - The agreed set of attributes a sparkplug must have
Thread Diameter, Thread Reach, Gap etc