QuakeSim Work: Web Services, Portlets, Real Time Data Services

Marlon Pierce (mpierce@cs.indiana.edu)Contributions: Ahmet Sayar, Galip Aydin, Mehmet Aktas, Harshawardhan Gadgil,

Zhigang Qi, Zao LiuCommunity Grids Lab

Indiana UniversityProject Funding: NASA AIST, ACCESS

QuakeSim Project Overview

• QuakeSim has been funded by CT, AIST, and ACCESS

• Collaboration of geophysicists and computer scientists to build cyber-infrastructure for geophysical research.

• CI research and development includes– Portlet-based portals

• AJAX enabled

– Geographical Information System services– Application services to run codes.

QuakeSim Project Development Overview

• Portlet-based portal components allow different portlets to be exchanged between projects.

– Form-based portlets --> Interactive Maps

– These are clients to Web services

– Share with collaborators of REASoN portal.

• Sensor Grid: Topic based publish-subscribe systems support operations on streaming data.

• Web services allow request/response style access to data and codes.

– GIS services (WMS, WFS)

– “Execution grid” services for running codes: RDAHMM, ST_Filter• Use application specific WSDL on top of generic code management

services.

– GPS daily archive Web Services provided by Scripps.

Evolving Project Philosophy

• Methodology for interoperability between portal projects exists today.– Portlets, web services– Scripps and IU are proving this.

• Must continue to move away from the portal priesthood and towards a “scientific mashup” model.– We still need to develop interesting services and client

libraries.– But we should make it easy for the application scientists to

• Add their own applications.• Add their own data.• Make their own web applications (ie mashups)

IU Portlet Development

We use JSR 168 portlets to build sharable portal plugins.

Portlet Summary

RDAHMM Set up and run RDAHMM, query Scripps GRWS GPS Service, maintain persistent user sessions.

ST_Filter Similar to RDAHMM portlet; ST_Filter has much more input.

Station Monitor Shows GPS stations on a Google Map, displays last 10 minutes of data.

Real Time RDAHMM Displays RDAHMM results of last 10 minutes of GPS data in a Google map.

Seismic Archive Query Portlet

Google Map portlet that shows seismic events based on your query. *

Fault Query Portlet Allows you to query the QuakeTables fault data base for information on faults.*

* Developed, needs to be converted into a portlet

RDAHMM Portlet: Main Navigation

RDAHMM Project Set Up

RDAHMM GRWS Query Interface

RDAHMM Results Page

Real Time RDAHMM Portlet

Station Monitor Portlet

ST_Filter Portlets

IU Web Service Work

Web ServicesAnt Execution Service General purpose service for running applications.

Wraps Apache Ant.

RDAHMM Service Based on Ant service, but provides a RDAHMM-specific API.

ST_Filter Service Runs ST_Filter, based on Ant service. Will have an ST_Filter-based WSDL.

Web Feature Service General purpose, OGC-compliant data service. Fault and seismic record data.

Web Map Service Map-generating service. We have built both a OGC WMS and a Google tiling-style map server (currently Indiana specific but can be generalized).

Context Service This is a lightweight metadata management service that can be used to store user session data. Project provenance: how did I make this output file?

Sensor Grid Overview

Publish/subscribe infrastructure for handling real time data.

Real-Time Services for GPS Observations

• Real-time data processing is supported by employing filters around publish/subscribe messaging system.

• The filters are small applications extended from a generic Filter class to inherit publish and subscribe capabilities.

Input Signal Output SignalFilter

Filter Chains

NaradaBrokering Topics

Real-Time positions on Google maps

Real-Time Station Position Changes

RDAHMM + Real-Time GPS Integration

SensorGrid Tests

Galip Aydin – Zhigang Qi

11/16/2006

SensorGrid Tests

• Two Major Goals: System Stability and Scalability– Ensuring stability of the Filter Services for

continuous operation.– Finding the maximum number of publishers

(sensors) and clients that can be supported with a single broker.

– Investigate if system scales for large number of sensors and clients.

Test Methodology

• The test system consists of a NaradaBrokering server and a three-filter chain for publishing, converting and receiving RYO messages.

• We take 4 timings for determining mean end-to-end delivery times of GPS measurements.

• The tests were run at least for 24 hours.

Ttransfer = (T2 – T1) + (T4 – T3)

1- System Stability Test• The basic system with

three filters and one broker.

• The average transfer time shows the continuous operation does not degrade the system performance.

System Stability Test

0

1

2

3

4

5

6

0:00 1:30 3:00 4:30 6:00 7:30 9:0010:3012:0013:3015:0016:3018:0019:3021:0022:30

Time of the Day

Time (ms)

Transfer Time Standard Deviation

2 – Multiple Publishers Test

• We add more GPS networks by running more publishers.

• The results show that 1000 publishers can be supported with no performance loss. This is an operating system limit.

Multiple Publishers Test

0

1

2

3

4

5

6

0:00 1:30 3:00 4:30 6:00 7:30 9:0010:3012:0013:3015:0016:3018:0019:3021:0022:30

Time of the Day

Time (ms)

Transfer Time Standard Deviation

3 – Multiple Clients Test

• We add more clients by running multiple Simple Filters.• The system can support as many as 1000 clients with very

low performance decrease.

Multiple Subscribers Test

0

5

10

15

20

25

30

35

40

0:00 1:30 3:00 4:30 6:00 7:30 9:0010:3012:0013:3015:0016:3018:0019:3021:0022:30

Time Of the Day

Time (ms)

Transfer Time Standard Deviation

Adding clients

1000 Clients

Extending Scalability

• The limit of the basic system appears to be 1000 clients or publishers.

• This is due to an Operating System restriction of open file descriptors.

• To overcome this limit we create NaradaBrokering networks with linking multiple brokers.

• We run 2 brokers to support 1500 clients.– Number of brokers can be increased indefinitely, so we can

potentially support any number of publishers and subscribers.

– Still have to test, of course.

4 – Multiple Brokers Test• Messages published

to first broker can be received from the second broker.

• We take timings on each broker.

• The results show that the performance is very good and similar to single broker test.

4 – Multiple Brokers Test

Multiple Broker Test Broker 1

0.00

5.00

10.00

15.00

20.00

25.00

30.00

35.00

0:001:303:004:306:007:309:0010:3012:0013:3015:0016:3018:0019:3021:0022:30Time Of The Day

Tim (ms)

Transit Time Standard Deviation

750 Clients

Multiple Broker Test Broker 2

0.00

5.00

10.00

15.00

20.00

25.00

30.00

35.00

40.00

0:001:30 3:004:306:00 7:309:00 10:3012:0013:3015:0016:3018:0019:3021:0022:30Time Of The Day

Time (ms)

Transfer Time Standard Deviation

750 Clients

Test Results

• The RYO Publisher filter publishes 24-hour archive of the CRTN_01 GPS network which contains 9 GPS stations.

• The single broker configuration can support 1000 clients or networks (9000 stations)

• The system can be scaled up by creating NaradaBrokering broker networks.

• Message order was preserved in all tests.

Federating Map Servers

Zao Liu, Marlon Pierce, Geoffrey Fox

Community Grids LaboratoryIndiana University

Integrating Map Servers

• Geographical Information Systems combine online dynamic maps and databases.

• Many GIS software packages exist• GIS servers around state of Indiana

– ESRI ArcIMS and ArcMap Server (Marion, Vanderburgh, Hancock, Kosciusco, Huntington, Tippecanoe)

– Autodesk MapGuide (Hamilton, Hendricks, Monroe, Wayne)

– WTH Mapserver™ Web Mapping Application (Fulton, Cass, Daviess, City of Huntingburg) based on several Open Source projects (Minnesota Map Server)

• Challenge: make 17 different county map servers from different companies work together.

– 92 counties in Indiana, so potentially 92 different map servers.

Considerations• We assume heterogeneity in GIS map and feature servers.

– GIS services are organized bottom-up rather than top-down.– Local city governments, 92 different county governments, multiple

Indiana state agencies, inter-state (Ohio, Kentucky) consideration, federal government data providers (Hazus).

– Must find a way to federate existing services.• We must reconcile ESRI, Autodesk, OGC, Google Map, and

other technical approaches.– Must try to take advantage of Google, ESRI, etc rather than compete.

• We must have good performance and interactivity. – Servers must respond quickly--launching queries to 20 different map

servers is very inefficient.– Clients should have simplicity and interactivity of Google Maps and

similar AJAX style applications.

Caching and Tiling Maps

• Federation through caching: – WMS and WFS resources are queried and results are stored on the cache

servers.– WMS images are stored as tiles.

• These can be assembled into new images on demand (c. f. Google Maps).

• Projections and styling can be reconciled.• We can store multiple layers this way.

– We build adapters that can work with ESRI and OGC products; tailor to specific counties.

• Serving images as tiles– Client programs obtain images directly from our tile server.

• That is, don’t go back to the original WMS for every request.– Similar approaches can be used to mediate WFS requests.– This works with Google Map-based clients.– The tile server can re-cache and tile on demand if tile sections are missing.

Map Server Example

Marion and Hancock county parcel plots and IDs are overlaid on IU aerial photographic images that are accessed by this mashup using Google Map APIs.

We cache and tile all the images from several different map servers. (Marion and Hancock actually use different commercial software.)