Date post: | 28-Nov-2014 |
Category: |
Technology |
Upload: | safe-software |
View: | 729 times |
Download: | 2 times |
CONNECT. TRANSFORM. AUTOMATE.
Managing Data Synchronization Jerrod Stutzman GIS Platform Engineer – Devon Energy
Project Goal
CONNECT. TRANSFORM. AUTOMATE.
! Create a centralized spatial system for storing and displaying SCADA (supervisory control and data acquisition) data
! View data on map via desktop and mobile ! Google-like search and performance
Challenges
! Need to see consistent data on desktop web app and custom iPhone app
! Location information from SCADA database is inconsistent and unreliable – need to join with reliable data for mapping
! Need to show on map along with other Devon assets (wells, pipelines, leases, etc) ! ArcSDE is authoritative spatial database
! Won’t be used if it isn’t fast!
Solution
! Spatial Reasoning System (SRS) ! OpenGeo Suite (POSTGIS data served via
GeoServer using GeoWebCache) ! Apache SOLR search platform ! Internally developed web and mobile apps ! FME & FME Server for data creation and
synchronization ! Custom code for certain integrations
Why OpenSource
! PERFORMANCE ! Biggest pain point historically is our well dataset.
Over 4 million features + 3 geometries (surface location, bottom location, directional path)
! POSTGIS + GeoServer render and cache this dataset quite fast
! POSTGIS can store all 3 geometries on one row! ! Reduced database complexity ! Low cost
! FME is used within the SRS to create and sync data for wells, pipelines, leases, seismic, facilities, etc
! FME Server handles scheduling and failure reporting (email)
SRS Integrations Diagram
Selected Workbench Example 1 EXCEL Spreadsheet to POSTGIS
Selected Workbench Example 1 EXCEL Spreadsheet to POSTGIS
! SCADA Radio Tower data is maintained in Excel ! Updated quarterly. Full dataset update ! GPS Location and Range need to be shown
! Need to store 1 to 3 geometries for each tower
Selected Workbench Example 1 EXCEL Spreadsheet to POSTGIS
Selected Workbench Example 2 Non-spatial Oracle to POSTGIS
Selected Workbench Example 2 Non-Spatial Oracle to POSTGIS
! Non-spatial well production data from Oracle DB is synchronized with POSTGIS
! Workflow:
Retrieve last run time from
POSTGIS Query
ORACLE for changes
Query POSTGIS for
matches Compare to determine
INSERT/UPDATE
Determine DELETES
Update POSTGIS
DB triggers update SOLR
Selected Workbench Example 2 Non-Spatial Oracle to POSTGIS
Selected Workbench Example 3 Seismic Data from Oracle DB
Selected Workbench Example 3 Seismic Data from Oracle DB
! Seismic data can be represented spatially as lines or polygons ! This is stored in the seismic DB as a coordinate list
of vertices (numbers only, non-spatial DB) ! This data needs to be ‘spatialized’ and stored in a
spatial database – synced nightly (incremental updates)
! Complexity Reduction – store all data (2 different geometry types) in same table
Selected Workbench Example 3 Seismic Data from Oracle DB
Result
Future
! Ability to do an “UPSERT” DB transaction would greatly simplify some of the logic required to determine DB transaction! ! ARCSDE30 Writer actually had this capability, if it
did not find the record to update, it would insert ! Need to revise old workbenches to use best
practices and gain efficiencies ! Scaling SRS to handle more data and users
Thank You!
! Questions?
! For more information: ! Jerrod Stutzman ! [email protected] ! Devon Energy Corporation
CONNECT. TRANSFORM. AUTOMATE.