Date post: | 01-Jan-2016 |
Category: |
Documents |
Upload: | sheryl-walters |
View: | 215 times |
Download: | 0 times |
11
Grids for Real-time and Streaming Applications
GCC2005 Beijing China
December 1 2005
Geoffrey Fox
Computer Science, Informatics, Physics
Pervasive Technology Laboratories
Indiana University Bloomington IN 47401
http://grids.ucs.indiana.edu/ptliupages/presentations/[email protected] http://www.infomall.org
22
Four Data Streaming Application Areas Data Assimilation applied to link the data deluge
(satellites, sensors, seismometers) in real time to small and large scale parallel simulations• Use in Earthquake Science
Department of Defense (and Homeland Security) have built the Global Information Grid with a target architecture NCOW (Network Centric Operations and warfare)• They submit no jobs; rather stream data to brokers from
which they are filtered and distributed• Includes their rather dated distributed simulation HLA
Audio-Video Conferencing implemented with services and Grid messaging
Hand-held Grid linking PDA/cell-phones to Grids
3
Database
SS
SS
SS
SS
SS
SS
SS
SS
SS
SS
FS
FS
FS
FS
FS
FS
FS
FS FS
FS
FS
FS
FS
FS
FS
FS
FS FS
FS
FS
PortalFS
OS
OS
OS
OS
OS
OS
OS
OS
OS
OS
OS
OS
MD
MD
MD
MD
MD
MD
MD
MD
MD
MetaDataFilter Service
Sensor Service
OtherService
AnotherGrid
Raw Data Data Information Knowledge Wisdom
Decisions
SS
SS
AnotherService
AnotherService
SSAnother
Grid SS
AnotherGrid
SS
SS
SS
SS
SS
SS
SS
SS
FS
SOAP Messages
44
Key Concepts Grid of Grids (System of Systems) allows “library”
approach to composing Grids Service Oriented architectures (Web or Grid services)
are attractive for many/most distributed systems There are many applications that are NOT best
considered as jobs and files (classic Grid) but rather as streams and filters (services)
Services exchanging messages becomesServices exchanging streams (sets of messages)
Publish-Subscribe messaging gives better QoS and management than point to point messaging with negligible performance loss
Always use standards including those for GIS
66
GIS and Sensor Grids OGC has defined a suite of data structures and services
to support Geographical Information Systems and Sensors
GML Geography Markup language defines specification of geo-referenced data
SensorML and O&M (Observation and Measurements) define meta-data and data structure for sensors
Services like Web Map Service, Web Feature Service, Sensor Collection Service define services interfaces to access GIS and sensor information
Grid workflow links services that are designed to support streaming input and output messages
We are building Grid (Web) service implementations of these specifications for NASA’s SERVOGrid
77
Grid Eventing via NaradaBrokering The Scripps Orbit and Permanent Array Center
(SOPAC) GPS station network data published in RYO format is converted to ASCII and GML
8
Grid Messaging with Grid of Grids Architecture
HPSearchManages
NaradaBrokering
Sensor Grid
WS-ContextStores dynamic data
WFS (GIS data)
Post beforeProcessing
Post afterProcessing
Notify
SubscribeDatabaseArchives
Web Feature Service
GIS Grid
GeographicalInformation System
DoD Principles
Filter orDatamining Grid
1010
WMS uses WFS that uses data sources
Railroads
RiversBridges
Interstate Highways
90
WFS Server
SQL Query
Railroads
[a-b]
SQ
L Q
uery
Riv
er [a
-d]
Bri
dge
[1-5
]
SQL QueryHigway [12-18]
`
ClientWMS
GetFeature
FeatureCollection
Get
Feat
ure
Feat
ureC
olle
ctio
n
<gml:featureMember> <fault> <name> Northridge2 </name> <segment> Northridge2
</segment> <author> Wald D. J.</author> <gml:lineStringProperty> <gml:LineString
srsName="null"> <gml:coordinates>
-118.72,34.243 -118.591,34.176 </gml:coordinates>
</gml:LineString> </gml:lineStringProperty> </fault> </gml:featureMember>
11
Google Map Client
Google Central
Google Map Client
UDDI
WFS2
Databases withSERVOGrid Faults
WFS1
SOAP
Sensor Grid
HTTP
Helper Services
Archived Real Time
DoD and Homeland Security can in a crisis combine custom geo-referenced data with that available from hundreds of thousands of computers from Microsoft, Yahoo and Google Just build simple services using Interoperability standards!
12
Real Time GPS and Google Maps
Subscribe to live GPS station. Position data from SOPAC is combined with Google map clients.
Select and zoom to GPS station location, click icons for more information.
13
Google maps can be integrated with Web Feature Service Archives to filter and browse seismic records.
Integrating Archived Web
Feature Services and Google Maps
15
Google plus GIS Grid Integrated with Los Alamos Critical Infrastructure Simulations
for DHS
Natural Gas Layer
Energy Power Layer
1616
NaradaBrokering 2003-2006 Messaging infrastructure for collaboration, peer-to-peer and Grids
Implements JMS and native high-performance protocols (message transit time of 1 to 2 ms per hop)
Order-preserving message transport with QoS and security profiles Support for different underlying transport such as TCP, UDP,
Multicast, RTP SOAP message support and WS-Eventing, WS-RM and WS-Reliability.
• WS-Notification when specification agreed Active replay support: Pause and Replay live streams. Stream Linkage: can link permanently multiple streams – using in
annotation of real-time video streams Replicated storage support for fault tolerance and resiliency to storage
failures. Management: HPSearch Scripting Interface to streams and brokers
(uses WS-Management) Broker Topics and Message Discovery: Locate appropriate Integration with Axis2 Web Service Container (?) High Performance Transport supporting SOAP Infoset
1717
0
1
2
3
4
5
6
7
8
9
100 1000
Tra
nsit
Del
ay
(Mill
isec
onds
)
Message Payload Size (Bytes)
Mean transit delay for message samples in NaradaBrokering: Different communication hops
hop-2
hop-5 hop-7
hop-3
Pentium-3, 1GHz, 256 MB RAM100 Mbps LAN
JRE 1.3 Linux
18
Average Video Delays for one broker – Performance scales proportional to number of brokers
Latency ms
# Receivers
One sessionMultiple sessions
30 frames/sec
19
Collaboration GridServices
UDDINaradaBroker
HPSearch
WS-Context
Gateway
WS-Security
NaradaBroker
NaradaBroker
Gateway
Gateway
Gateway
XGSP MediaService
Video Mixer
Transcoder
Audio Mixer
Replay
Record
Annotate
Thumbnail
WhiteBoard
SharedDisplay
SharedWS
2121
e - Annotation Player
Archived stream player Annotation / WB
player
Archieved stream list
Real time stream list
e - Annotation Whiteboard
Real time stream player Archived Real Time Real Time
Stream List Stream List Player
e-Annotation Archived Stream Annotated e-Annotation
Player Player Stream Player Whiteboard
2222
Location of software for Grid Projects in Community Grids Laboratory
htpp://www.naradabrokering.org provides Web service (and JMS) compliant distributed publish-subscribe messaging (software overlay network)
htpp://www.globlmmcs.org is a service oriented (Grid) collaboration environment (audio-video conferencing)
http://www.crisisgrid.org is an OGC (open geospatial consortium) Geographical Information System (GIS) compliant GIS and Sensor Grid (with POLIS center)
http://www.opengrids.org has WS-Context, Extended UDDI etc.
The work is still in progress but NaradaBrokering is quite mature
All software is open source and freely available