Dvoy Networking Ideas
OpenGIS Web Services
• Mission: Definition and specification of geospatial web services.• A Web service is an application that can be published, located, and dynamically
invoked across the Web.• Applications and other Web services can discover and invoke the service.• The sponsors of the Web services initiative include
– Federal Geographic Data Committee– Natural Resources Canada– Lockheed Martin– National Aeronautics and Space Administration– U.S. Army Corps of Engineers Engineer Research and Development Center– U.S. Environmental Protection Agency EMPACT Program– U.S. Geological Survey– US National Imagery and Mapping Agency.
• Phase I - February 2002 – Common Architecture: OGC Services Model, OGC Registry Services, and Sensor
Model Language. – Web Mapping: Map Server- raster, Feature Server-vector, Coverage Server-image,
Coverage Portrayal Services. – Sensor Web: OpenGIS Sensor Collection Service for accessing data from a variety of
land, water, air and other sensors.
D-DADS Architecture
ARC/INFO
VirtualDataCube
ArcSDETranslator
OLAPService
Provider
DataCube
LegacyDatabase
CustomOLAP
Translator
DataCube
SQLDatabase
OLAP ServiceProvider
GISTable
OLAP
StandardizedDescription &
Format
Database(SQL,
Oracle,etc.)
ArcSDETranslator
GISMap
DataProviders
Data Access andManipulation Tools
UserInteraction
DataCube
ArcIMS
The D-DADS Components
• Data Providers supply primary data to system, through SQL or other data
servers. • Standardized Description & Format populate and describe the data
cubes and other data types using a standard metadata describing data
• Data Access and Manipulation tools for providing a unified interface to data cubes, GIS data layers, etc. for accessing and processing (filtering, aggregating, fusing) data and integrating data into virtual data cubes
• Users are the analysts who access the D-DADS and produce knowledge from
the data
The multidimensional data access and manipulation component of D-DADS will be implemented using OLAP.
Interoperability
“the ability to freely exchange all kinds of spatial information about the Earth and about objects and phenomena on, above, and below the Earth’s surface; and to cooperatively, over networks, run software capable of manipulating such information.” (Buehler & McKee, 1996)
Such a system has two key elements:
• Exchange of meaningful information
• Cooperative and distributed data management
One requirement for an effective distributed environmental data system is interoperability, defined as,
On-line Analytical Processing: OLAP
• A multidimensional data model making it easy to select, navigate, integrate and explore the data.
• An analytical query language providing power to filter, aggregate and merge data as well as explore complex data relationships.
• Ability to create calculated variables from expressions based on other variables in the database.
• Pre-calculation of frequently queried aggregated values, i.e. monthly averages, enables fast response time to ad hoc queries.
User Interaction with D-DADS
Query
Data View(Table, Map, etc.)
Distributed Database
XML data
XML data
Metadata Standardization
• The Supersite Data Management Workgroup
• NARSTO
• FGDC
Metadata standards for describing air quality data are currently being actively pursued by several organizations, including:
Potential D-DADS Nodes
The following organizations are potential nodes in a distributed data analysis and dissemination system:
• CAPITA
• NPS-CIRA
• EPA Supersites- California- Texas- St. Louis
Summary
In the past, data analysis has been hampered by data flow resistances. However, the tools and framework to overcome each of these resistances now exist, including:
• World Wide Web• XML• OLAP• OpenGIS• Metadata standards
Incorporating these tools will initiate a distributed data analysis and dissemination system.
‘Global’ and ‘Local’ AQ Analysis
• AQ data analysis needs to be performed at both global and local levels
• The ‘global’ refers to regional national, and global analysis. It establishes the larger-scale context.
• ‘Local’ analysis focuses on the specific and detailed local features
• Both global and local analyses are needed for for full understanding.
• Global-local interaction (information flow) needs to be established for effective management.
National and Local AQ Analysis
Data Re-Use and Synergy
• Data producers maintain their own workspace and resources (data, reports, comments).
• Part of the resources are shared by creating a common virtual resources.
• Web-based integration of the resources can be across several dimensions:Spatial scale: Local – global data sharing
Data content: Combination of data generated internally and externally
• The main benefits of sharing are data re-use, data complementing and synergy.
• The goal of the system is to have the benefits of sharing outweigh the costs.
Content
Content
User
User
User
LocalLocal
GlobalGlobal
Virtual Shared Resources
Data, KnowledgeTools, Methods
User
User
Shared part of resources
Integration for Global-Local Activities
Global Activity Local Benefit
Global data, tools Improved local productivity
Global data analysis Spatial context; initial analysis
Analysis guidance Standardized analysis, reporting
Local Activity Global Benefit
Local data, tools Improved global productivity
Local data analysis Elucidate, expand initial analysis
Identify relevant issues Responsive, relevant global analysis
Global and local activities are both needed – e.g. ‘think global, act local’
‘Global’ and ‘Local’ here refers to relative, not absolute spatial scale
Content Integration for Multiple Uses (Reports)
Data from multiple measurements are shared by their providers or custodiansData are integrated, filtered, aggregated and fused in the process of analysisReports use the analysis for Status and Trends; Exposure Assessment; Compliance …
The creation of the needed reports requires data sharing and integration from multiple sources.
Federated Data Warehouse Features
• As much as possible, data should reside in their respective home environment. ‘Uprooted’ data in decoupled databases tend to decay i.e. can not be easily updated, maintained, enriched.
• Data Providers would need to ‘open up’ their SQL data servers for limited data subsets and queries, in accordance with a ‘contract’. However, the data structures of the Providers will not need to be changed.
• Data from the providers will be transferred to the ‘federated data warehouse’ through (1) on-line DataAdapters, (2) Manual web submission and (3) Semi-automated transfer from the NARSTO archive.
• Retrieval of uniform data from the data warehouse facilitates integration and comparison along the key dimensions (space, time, parameter, method)
• The open architecture data warehouse (see Web Services) promotes the building of further value chains: Data Viewers, Data Integration Programs, Automatic Report Generators etc..
Navigation Service
DVoy: Components and Data Flow
Legacy Data
Publish
(DataSet)
DataSet Records
• Provider Descript.
• Service Descript.
• Measure Access
Find (Measure)
DataSet Recs
Selected Measure Record
• Dvoy
Data Wrapping
WebServiceBind (Measure, FocusCube)
Data Delivery
WebServiceFocusCube, GlobCursor,
Measure, Granule
Time Chart
Layered Map
DataToView
DataForCursorAndView
Catalog Service Data Services
Data Delivery
WebService
Presentation Services
Data Focus Range
Rendering
Cursor
Viewer Layers
Dim1: LonDim2: Lat
Data provided by each dimension of a View:
Dim1.Type, Dim1.Min, Dim1.Max
Dim2.Type, Dim2.Min, Dim2.Max
….
Current Dim.Types:
Latitude, Longitude, DateTime, Elevation
Federated Data Services Architecture
XML WebServices
Satellite
Vector
GIS Data
XDim Data
OLAPCube
SQLTable
HTTPServices
Text Data
WebPage
TextData
Scatter Chart
Text, Table
Data View & Process TierLayered Map
Cursor
Data Warehouse Tier
Data View
Manager
Connection
Manager
Data Access
Manager
Cursor-Query
Manager
OpenGISServices
Data are rendered by linked Data Views (map, time, text)
Distributed data of multiple types (spatial, temporal text)
The Broker handles the views, connections, data access, cursor
Time Chart
Dvoy Federated Information System
• Dvoy offers a homogeneous, read-only access mechanism to a dynamically changing collection of heterogeneous, autonomous and distributed information sources.
• Data access uses a global multidimensional schema consisting of spatial, temporal and parameter dimensions
• The uniform global schema is suitable for data browsing and online analytical processing, OLAP
• The limited global query capabilities yield slices along the spatial, temporal and parameter dimensions of the multidimensional data cubes.
Architecture of DATAFED Federated Data SystemAfter Busse et. al., 1999
• The main software components of Dvoy are wrappers, which encapsulate sources and remove technical heterogeneity, and mediators, which resolve the logical heterogeneity.
• Wrapper classes are available for geo-spatial (incl. satellite) images, SQL servers, text files,etc. The mediator classes are implemented as web services for uniform data access to n-dimensional data.
Integration Architecture (Ullman, 1997)
• Heterogeneous sources are wrapped by software that translates between the sources local language, model and concepts and the shared global concepts
• Mediators obtain information from one or more components (wrappers or other mediators) and pass it on to other mediators or to external users.
• In a sense, a mediator is a view of the data found in one or more sources; it does not hold the data but it acts as it it did. The job of the mediator is to go to the sources and provide an answer to the query.
Federated PM and Haze Data Warehouse Project
a sub- project of
(enter your sticker & logo here )
Nov 20, 2001, RBH
St. Louis Midwest Supersite Project
Regional Planning OrganizationRPO
EPA Supersites SupSite
NARSTO PMNARSTO
EPA Division1, Division2, Division2 EPA
Me and my dog for our aerosol projectMe
PM/Haze Data Flow in Support of AQ Management
• There are numerous organizations in need of data relevant to PM/Haze
• Most interested parties (stakeholders) are both producers and consumers of PM and haze data
• There is a general willingness to share data but the resistances to data flow and processing are too high
RPO
RPO
RPO
Regional Planning Orgs
FLM
FLM
FLM
Federal Land Managers
EPA
EPA
EPA
EPA Regul. & Research
Industry
AcademicNARSTO
Other: Private, Academic
SuperSite
Shared PM/Haze
Data
• PM and haze data are used for may parts of AQ management, mostly in form of Reports
• The variety of pertinent (ambient, emission) data come from many different sources
• To produce relevant reports, the data need to be ‘processed’ (integrated, filtered aggregated)
Scientific and Administrative Rationale for Resource Sharing
Scientific Rationale:• Regional haze and its precursors have a 1000-10000 km airshed.• (Smoke, Dust, Haze) – Data integration • Substantial fraction of haze originates from natural sources or
from out-of-jurisdiction man-made sources• Cross-RPO data and knowledge sharing yields better operational
and science support to AQ management
Management Rationale:
• Haze control within some RPOs cannot yield • Data sharing saves money and ….
A Strategy for the Federated PM/Haze Data Warehouse
• Negotiate with the data providers ‘open up’ their data servers for limited, controlled, access in accordance with clear ‘access contract’ with the Federated Warehouse
• Design an interface to the warehoused datasets that has simple data access and satisfies the data needs of most integrating users.(oxymoron ????)
• Facilitate the the development of shared value-adding processes (analysis tools, methods) that refine the raw data to useful knowledge
Three-Tier Federated Data Warehouse Architecture
(Note: In this context, ‘Federated’ differs from ‘Federal’ in the direction of the driving force. Federated meant to indicate a driving force for sharing from ‘bottom up’ i.e. from the members, not dictated from ‘above’, by the Feds)
1. Provider Tier: Back-end servers containing heterogeneous data, maintained by the federation members
2. Proxy Tier: Retrieves designated Provider data and homogenizes it into common, uniform Datasets
3. User Tier: Accesses the Proxy Server and uses the uniform data for presentation, integration or processing
Provider Tier
Heterogeneous data in distributed SQL Servers
Proxy Tier
Data homogenization, transformation
Federated Data Warehouse
User Tier
Data presentation, processing
Federated Data Warehouse Interactions
• The Provider servers interact only with the Proxy Server in accordance with the Federation Contract– The contract sets the rules of interaction (accessible data subsets, types of queries)– Strong server security measures enforced, e.g. through Secure Socket layer
• The data User interacts only with the generic Proxy Server using flexible Web Services interface– Generic data queries, applicable to all data in the Warehouse (e.g. data sub-cube by space, time, parameter)– The data query is addressed to the Web Service provided by the Proxy Server– Uniform, self-describing data packages are passed to the user for presentation or further processing
SQLDataAdapter1
CustomDataAdapter
SQLDataAdapter2
SQLServer1
SQLServer2
LegacyServer
Presentation
Data Access & Use
Provider Tier Heterogeneous Data
Proxy Tier
Data Homogenization, etc.
Member ServersProxy Server
User Tier
Data Consumption
Processing
Integration
Federated Data Warehouse
Fire Wall, Federation ContractWeb Service, Uniform Query & Data
‘Global’ and ‘Local’ AQ Analysis
• AQ data analysis needs to be performed at both global and local levels
• The ‘global’ refers to regional national, and global analysis. It establishes the larger-scale context.
• ‘Local’ analysis focuses on the specific and detailed local features
• Both global and local analyses are needed for for full understanding.
• Global-local interaction (information flow) needs to be established for effective management.
National and Local AQ Analysis
Data Model Ray Plante, Virtual Obs
• What’s the difference between Data Models and Metadata? Intertwined– metadatum: a datum with a name or semantic tag that refers to the data– data model: a description of the relationships between metadata
• structural & logical relationships between compound objects & their components• operations that can be performed on them (really -
– framework: the architecture/process used to define metadata/data models that enables their ready use in applications
• Formalized data modeling process– encourages community involvement for defining standard models & metadata– structure enables easy verification, dissemination, & automated use– “standard” metadata should point directly to components of the “standard”
models– allow groups to define metadata independent of a “standard” metadata
• Practical Difference?– data model captures as complete a picture of a concept as possible– metadata represents the instantiation of portion of the model’s components
• Data access through a data model (Wrapper Classes for each data model)
Integration for Global-Local Activities
Global Activity Local Benefit
Global data, tools => Improved local productivity
Global data analysis => Spatial context; initial analysis
Analysis guidance => Standardized analysis, reporting
Local Activity Global Benefit
Local data, tools => Improved global productivity
Local data analysis => Elucidate, expand initial analysis
Identify relevant issues => Responsive, relevant global work
Global and local activities are both needed – e.g. ‘think global, act local’
‘Global’ and ‘Local’ here refers to relative, not absolute scale
Federated Data System Features
• Data reside in their respective home environment where it can mature. ‘Uprooted’ data in centralized databases are not easily updated, maintained, enriched.
• Abstract (universal) query/retrieval facilitates integration and comparison along the key dimensions (space, time, parameter, method)
• The open data query based on Web Services promotes the building of further value chains: Data Viewers, Data Integration Programs, Automatic Report Generators etc..
• The data access through the Proxy server protects the data providers and the data users from security breaches, excessive detail
Integration for Global-Local Activities
Global Activity Local Benefit
Global data & analysis Spatial context; initial analysis
Analysis guidance Standardized analysis, reporting
Local Activity Global Benefit
Local data & analysis Elucidate, expand initial analysis
Identify relevant issues Responsive, relevant global analysis
Global and local activities are both needed – e.g. ‘think global, act local’
‘Global’ and ‘Local’ here refers to relative, not absolute spatial scale
Data Re-Use and Synergy
• Data producers maintain their own workspace and resources (data, reports, comments).
• Part of the resources are shared by creating a common virtual resources.
• Web-based integration of the resources can be across several dimensions:Spatial scale: Local – global data sharing
Data content: Combination of data generated internally and externally
• The main benefits of sharing are data re-use, data complementing and synergy.
• The goal of the system is to have the benefits of sharing outweigh the costs.
Content
Content
User
User
User
LocalLocal
GlobalGlobal
Virtual Shared Resources
Data, KnowledgeTools, Methods
User
User
Shared part of resources
Federated Information System
• Providers maintain their own workspace and resources (data, tools, reports)
• Part of the private resources are exposed as shared (federated) resources
• The Federation facilitates finding, accessing and usage of the shared resources
Data sharing federations: • Open GIS Consortium (GIS data layers)• NASA SEEDS network (Satellite data)• NSF Digital Government • EPA’s National Env. Info Exch. Network.
InfoData
Shared (Federated) Resources
Data, Services, Tools, Methods
SharedPrivate
Other Federations
Providers/Users
Data Federation Concept and the FASNET Network
Schematic representation of data sharing in a federated information system.Based on the premise that providers expose part of their data (green) to others
Schematics of the value-adding network proposed for FASTNETComponents embedded in the federated value network
Data Acquisition and Usage Value Chain
Monitor StoreData 1
Monitor StoreData 2
Monitor StoreData n
Monitor StoreData m
IntData1
IntDatan
IntData2 Virtual Int. Data
Processes of the Information Value Chain
•
Informing Knowledge
ActionProductive Knowledge
Information
Organizing
Grouping Classifying Formatting Displaying
Analyzing
SeparatingEvaluating
Interpreting
Synthesizing
Judging
Options Quality Advantages
Disadvantages
Deciding
Matching goals, Compromising
Bargaining Deciding
CIRA VIEWS Langley IDEA AQ ManagerWG Summary Rpt
Data
(after Taylor, 1975)
Examples:
Data Flow and Processing
Next Process
Next Process
Why? How?
When? Where?CATT: A Community Tool!
Part of an Analysis Value Chain
Aerosol Data
Collection IMP. EPA
Aerosol Sensors
Integration VIEWS
Integrated AerData
AEROSOL
Weather Data
Assimilate NWS
Gridded Meteor.
Trajectory ARL
Traject.Data
TRANSPORT
TrajData Cube
Aggreg. Traject.
AerData Cube
CATT
Aggreg.Aerosol
CATT-In CAPITA
CATT-In CAPITA
There!
Not There! Further Analysis
GIS
Grid Processing
Emission
Comparison
Fast Aerosol Sensing Tools for Natural Event Tracking FASTNET
Analysts Console
Community Website
Distributed Programming: Interpreted and Compiled
• Web services allow processing of distributed data– Data are distributed and maintained by their custodians,
– Processing nodes (web-services) are also distributed
– ‘Interpreted’ web-programs for data processing can be created ad hoc by end users
• However, ‘interpreted’ web programs are slow, fragile and uncertain– Slow due to large data transfers between nodes
– Fragile due to instability of connections
– Uncertain due to failures of data provider and processing nodes
• One solution is to ‘compile’ the data and processing services– Data compilation transforms the data for fast, effective access (e.g. OLAP)
– Web service compilation combines processes for effective execution
• Interpreted or compiled?– Interpreted web programs are simpler and up to date but slow, fragile, uncertain
– Compiled versions are more elaborate and latent but also faster and more robust
– Frequently used datasets and processing chains should be compiled and kept current
Interpreted and Compiled Service
Point Access
Point Grid
Grid Render
Point Render
PtGrid Overla
y
Point Access
Point Grid
Grid Render
Point Access
Point Render
PtGrid Overlay
Interpreted Service
• Processes distributed
• Data flow on Internet
Compiled Service
• Processes in the same place
• Data flow within aggregate service
• Controllers, e.g. zoom can be shared
Data Flow
Control Flow
Voyager: The Program
• The Voyager program consists of a stable core and adoptive input/output section• The core executes the data selection, access portrayal tasks• The adoptive, abstract I/O layer connects the core to evolving web data, flexible
displays and to the a configurable user interface:– Wrappers encapsulate the heterogeneous external data sources and homogenize the access– Device Drivers translate generic, abstract graphic objects to specific devices and formats – Ports expose the internal parameters of Voyager to external controls
Data Sources
Controls
Displays
Voyager Core
Data Selection
Data Access
Data Portrayal
Adoptive Abstract I/O Layer
Dev
ice
Dri
vers
Ports
Wra
pp
ers
Dvoy_Services: Generic Software components
Webservice
Param 1
Param2
Service state
Webservice Adaptor
User Interface Module
Controller
state I/o ports
state I/o ports
Web service calls
Web serviceOutput data
Web serviceInput data
User Interface Module
UIM extracts relevant UI parameters from STATE
User changes UI parameters
UIM transmits modified UI parameters to STATE
Service Chain STATE Module
Contains the state params for all services in the chain
Has ports for getting/setting state params
Service Adopter Module
Gets input data from upsteam service
Gets service params from STATE
Make service call
Service Adopter ModuleGets input data from upsteam service
Gets service params from STATE
Make service call
Web service Service Module
Gets service call from Adopter module
Executes service
Returns output data
PointAccess->Grid->GridRender Service Chain
• The service chain interpreter make ONLY 2 sequential calls, stated in the data flow program:– GetMapPointDataAdaptor– RenderMapviewPoint Adaptor
GetMapPointDatadataset_abbr: IMPROVE
Param_abber SOILf
datatime: 2001-04-16
sql_filter:
RenderMapviewPoint
dataset_url:
output_format:
out_image_width:
Etc…..
Service state
GetMapPointData Adaptor
RenderMapviewPoint Adaptor
GetMapPointData Selector
RenderMapviewPoint Selector
state I/o ports
state I/o ports
Web service calls
Web serviceOutput data
Service state
PointAccess->Grid->GridRender Service Chain
• The service chain interpreter make ONLY 3 sequential calls, stated in the data flow program:– GetMapPointDataAdaptor– GridMapviewPointAdaptor– RenderMapviewGridAdaptor
GetMapPointDatadataset_abbr: IMPROVE
Param_abber SOILf
datatime: 2001-04-16
sql_filter:
RenderMapviewGriddataset_url:
output_format:
out_image_width:
Etc…..
GetMapPointData Adaptor
RenderMapviewGrid Adaptor
GetMapPointData Selector
RenderMapviewGrid Selector
GridMapviewPoint Selector
GridMapviewPointdataset_url:
output_format:
out_image_width:
Etc…..
GridMapviewPoint Adaptor
state I/o ports
state I/o ports
Web service calls
Web serviceOutput data
Service state Service state Service state
VOYAGER Web Services
Layered Map
Time ChartVector
GIS Data
XDim DataSQL Tables
WebImages
Publish, Find, Bind
Catalog, Data & Tools
Uniform Access
Scatter Chart
S u p p o r tCoordination T e c h n o l o g i e s
Users
Select, Overlay, Explore; Multidimensional data
Providers
Maintain distributed data; Heterogeneous coding , access
Voyager Web Services
Homogenize data access Catalog, access, transform data
C O M M U N I T Y
Services Program Execution:
Reverse Polish Notation
Writing the WS program:- Write the program on the command line of a URL call- Services are written sequentially using RPN - Replacements
Connector/Adaptor: - Reads the service name from the command line and loads its WSDL- Scans the input WSDL - The schema walker populates the service input fields from:
- the data on the command line - the data output of the upstream process - the catalog for the missing data
Service ExecutionFor each service
Reads the command line, one service at a timePasses the service parameters to the above Connector/Adopter, which prepares
the serviceExecutes the service
It also handles the data stack for RPN
Pg. 25-26; 23-24N2