USGS ShakeCastDelivering Earthquake Shaking Data

To The People Who Need It

Philip A. NaeckerChief ScientistGatekeeper Systems

pan@gatekeeper.com

Gatekeeper Systems ShakeCast Features and Architecture– June 2003 Slide 2

ShakeCast Features and Architecture Project Origins Project Team Issues with Current

Delivery of ShakeMaps Project Goals and

ShakeCast System Goals ShakeCast Terminology

ShakeCast Features and Functions

Prototype V1.0 Futures

ShakeCast Architecture Protocols and Transport Database Development

Environment

Gatekeeper Systems ShakeCast Features and Architecture– June 2003 Slide 3

Project Origins Success of ShakeMap Project

ShakeMaps are data rich Recurrent difficulties utilizing ShakeMaps

Consumers have had difficulty with network configuration, firewalls, and data processing

Each consumer wants a different configuration of feed, requiring “hand holding” and creating a maintenance morass

Missed opportunities for data utilization Significant risk that information system fragility

might impact successful application of ShakeMaps in a major event

Gatekeeper Systems ShakeCast Features and Architecture– June 2003 Slide 4

ShakeCast Project Team What eventually became the ShakeCast Team

has been in conversation about these issues since 1998

USGS: Dave Wald and Bruce Worden Peter German

Consulting geologist, seismic data processing expert Creator of the CUBE system

Robert Nigbor USC Professor Long involvement in practical applications of shaking

data (e.g., Intel)

Gatekeeper Systems ShakeCast Features and Architecture– June 2003 Slide 5

Software Development Expertise Gatekeeper Systems

In business since 1993 selling software for utilities Specialize in high performance, high availability

systems for large database, GIS, and Internet applications

Customers include large utilities and municipalities• City of LA, Santa Barbara, PacifiCorp, Las Vegas Valley

Water District, Georgia Power, Contra Costa County, etc.

Gatekeeper Systems ShakeCast Features and Architecture– June 2003 Slide 6

ShakeCast Software Development Team Bruce Worden – ShakeMap Software Development Phil Naecker

Civil Engineer, 10 years as engineering consultant focused on water and wastewater utilities

Built large high performance data management systems for business and engineering applications since 1971

Steve Caine 45 years experience building software Founded firm that sold critical software systems to DOD and

related agencies for over thirty years Internet and Open Systems Expert

Dave Burke 25 years experience building software Exceptional competence at building complex systems

Gatekeeper Systems ShakeCast Features and Architecture– June 2003 Slide 7

ShakeCast Problem and Solution Current Problems ShakeCast Solution

Current Delivery of ShakeMaps Custom solution for each organization

Unknown failure modes Not built by experts Not built using latest Internet technologies

Difficult to implement and maintain Significant work for USGS and consumer Not conducive to adding new data products or

technologies Each organization has home-grown notification

system (or none at all)

Gatekeeper Systems ShakeCast Features and Architecture– June 2003 Slide 9

EQ Systems 2002

SeismicProcessing

ShakeMapProcessing

InstrumentNetwork

Public/PrivateNetworks

PrivateNetworks

Archives

Users

AkamaiServers

PublicNetworks

ConsumerInformation

System

PrivateNetworks

FTP over PublicNetworks

Goals: ShakeCast Project Provide rapid and reliable delivery of information about

earthquake shaking to the people who need it Initial audience focuses on utilities, large facility owners,

municipalities, and news outlets “Shaking Information” is broadly defined

Satisfy some of the needs for post-event activities such as refinement of response plans and after-the-fact response assessment by accurately recording who knew what when

Deliver an open technology platform for earthquake information delivery that facility managers and the USGS can depend on and feel comfortable in building upon

Gatekeeper Systems ShakeCast Features and Architecture– June 2003 Slide 11

EQ Systems 2003

SeismicProcessing

ShakeMapProcessing

InstrumentNetwork

Public/PrivateNetworks

PrivateNetworks

Archives

Users

DistributionSystem

EventNotification

System

AkamaiServers

Processingand

Notification

PublicNetworks

ConsumerInformation

System

PrivateNetworksShakeCast

Gatekeeper Systems ShakeCast Features and Architecture– June 2003 Slide 12

Goals: Data Delivery and Notification Delivery and Notification Must Be Extremely

Reliable Multiple data sources Multiple data paths Robust testing protocols Set and Forget Design

Fast Under All Load Conditions Essentially no training required for proper

installation and maintenance

Gatekeeper Systems ShakeCast Features and Architecture– June 2003 Slide 13

Goals: Other System Attributes Firewall Friendly System Must Be Complete

Functional “Out of the Box” in a few hours Deliver data and most commonly-used tools

Extensible and Flexible Able to integrate with more advanced tools Customizable by organizations and individual users

Traceable and Auditable System and usage audit logs, consistent timestamps Objective and reproducible notification actions

Gatekeeper Systems ShakeCast Features and Architecture– June 2003 Slide 14

ShakeCast Features and Functions Phased rollout of features Functions provide infrastructure, not enforce

policy on how ShakeMaps are used Reliability and robustness are designed in from

the beginning

Gatekeeper Systems ShakeCast Features and Architecture– June 2003 Slide 15

ShakeCast Technology Plan Open source using widely used tools

Apache, Perl, Oracle, Windows, Internet Explorer Heavily documented code and databases Based on commodity technologies:

Internet and Web Email and SMS pagers Relational databases

Security and reliability designed in from the start

Gatekeeper Systems ShakeCast Features and Architecture– June 2003 Slide 16

ShakeCast Project Phases Phase 1 - Prototype Phase 2 – Reference System Phase 2 – Open Source

Gatekeeper Systems ShakeCast Features and Architecture– June 2003 Slide 17

ShakeCast Phase 1 - Prototype Funded by USGS Purposes are proof of concept, demonstration Implemented by Gatekeeper Systems and Bruce

Worden Due Summer 2003 CalTrans will be first testbed organization SBC/PacBell anxious to be next testbed Basic functionality, no easy installation or

customization, single hardware/software platform

Gatekeeper Systems ShakeCast Features and Architecture– June 2003 Slide 18

ShakeCast Phase 2 – Reference System Shopping for funding now Fully installable (goal is one hour if the server

already has a web server and database) Easily configurable Multiple hardware/software platforms Looking for early adopters Expect network of dozens to hundreds of

systems and tens of thousands of end users

Gatekeeper Systems ShakeCast Features and Architecture– June 2003 Slide 19

ShakeCast Phase 3 – Open Source Shared development between user

organizations Enhancements to core system

Participation by universities Commercialization and Extension

Integration with internal systems Improved data for shaking estimates Readily available sources for fragility estimates

Hoping for thousands of systems and millions of end users

Gatekeeper Systems ShakeCast Features and Architecture– June 2003 Slide 20

ShakeCast Terminology Server: every ShakeCast machine is a server Upstream/Downstream: direction of data flow,

although complex network topologies are allowed Parameters: measures of shaking generated by

ShakeMap system Products: data files, in various formats and multiple

parameters, moved between ShakeCast machines Grid File: ShakeMap grids containing the raw

parameter data Notification: Detailed electronic message about a

specific event, system activity, or shaking level to a specific user or group of users

Gatekeeper Systems ShakeCast Features and Architecture– June 2003 Slide 21

ShakeCast Software Features Reliably and automatically receive and process shaking

data from ShakeMap Let organizations and users define locations of interest

(facilities) and set shaking thresholds (green, yellow, red) in multiple shaking metrics (acceleration, instrumental intensity, etc.)

Reliably deliver to end users electronic notification of facility damage estimates in a prioritized, customized, easy-to-use form

Make maps and reports from local servers available via the Web

Easily integrate with consumer’s other IT systems Provide for end-to-end testing and upgrades

Gatekeeper Systems ShakeCast Features and Architecture– June 2003 Slide 22

Prototype Features Receive, store and forward ShakeMaps and

associated metadata in a reliable manner Unpack ShakeMap grids into a relational

structure Notify users of shaking and ShakeCast activity

Email and pager System administrator, ShakeCast developer, and

end user events Produce detailed log files of ShakeCast and

user activity

Gatekeeper Systems ShakeCast Features and Architecture– June 2003 Slide 23

Prototype Features - Transport Receive from multiple upstream ShakeCast and

ShakeMap servers Versioned products HTTP push or HTTP polling/pull Can transport not only shaking data but system

metadata: i.e. product types, message types, etc. Basic filtering for ShakeMap feed

Bounding rectangle Peak grid values

Retry and basic error handling Basic test suites

Gatekeeper Systems ShakeCast Features and Architecture– June 2003 Slide 24

Prototype Features – Notification On events: map generated, errors, recall/cancel,

product delivery On ShakeMap parameters: magnitude,

acceleration, etc. On location-specific shaking for any ShakeMap

parameter On exceedence of facility fragility for any

ShakeMap parameter (green, yellow, red) Message format driven by easily customized

templates, includes direct Web links

Gatekeeper Systems ShakeCast Features and Architecture– June 2003 Slide 25

Prototype Features - Database Most configuration information, all ShakeMap

data, and all user data is stored in the database Support for both Access and Oracle Site administrator can access database using

standard SQL and other standard tools such as MS Access, Visual Basic, Perl, etc.

Gatekeeper Systems ShakeCast Features and Architecture– June 2003 Slide 26

Additional V1.0 Features Consolidated

notification Professional

documentation Complete installation

procedure with upstream registration

User configuration web pages

Full support for both Unix and NT

Deliver ShakeMap web pages locally

Enhanced error handling Enhanced test

procedures Call out from ShakeCast

to private scripts to invoke site-specific functions

Automated end-to-end testing

Gatekeeper Systems ShakeCast Features and Architecture– June 2003 Slide 27

Features Futures Richer notification options

Support for multiple related events More intelligent prioritization of messages More complicated notification logic Support for positive response (confirmation) Richer web links with active web pages to help

users manage large lists of facilities More database platforms certified Open source shared development environment Upstream reporting of ShakeCast usage, server

health and status, and test results

Gatekeeper Systems ShakeCast Features and Architecture– June 2003 Slide 28

ShakeCast Implementation Implementation Principles

Use open and familiar tools and protocols Follow path of least resistance for network and

security managers Open, readable, understandable source code

HTTP and HTTPS transport Relational database for all data storage NT Service/Unix Daemon for event loop

processing

Gatekeeper Systems ShakeCast Features and Architecture– June 2003 Slide 29

Software Development Environment NT and Unix (but could be anything) Perl for CGI (but could be anything) Standard SQL (and optional SQL-based tools) Source code management in CVS (Web-based

source management system designed for shared development)

No C, VB, Java or compiled languages required, providing short development cycles, transparency, portability

ShakeCast System Data Flow

ShakeCastServer

ShakeCastServer

ShakeCastServer

ShakeCastServer

GIS SystemsControl SystemsAlarm Systems

Data andInvocation

EmailSystem

PagingSystem

Users

NotificationMessages

Web PagesShakeCastData and XML

USGS Systems Private Systems

ShakeMapServer

ShakeMapServer

httphttp

http

http http

smtp

Email/PagerMessages

httpftp/NFS

Gatekeeper Systems ShakeCast Features and Architecture– June 2003 Slide 31

Protocols and Transport Architecture HTTP CGI scripts for response to ShakeCast

requests ShakeCast metadata encoded in simple XML HTTP GETs for file delivery Can easily use HTTPS if needed Authenticated server-server exchange:

MD5 secured passwords for authentication Separate passwords for each server pair

New CGI routines can be easily added Use Apache HTTP server, but could use others

Gatekeeper Systems ShakeCast Features and Architecture– June 2003 Slide 32

Database Architecture Fully normalized data model High cardinality transaction data can be purged

if necessary Shared data elements have shared primary keys Locally-generated data elements have locally-

generated primary keys Multiple database platforms possible

MS Access and Oracle currently supported Can easily be extended to use Oracle Spatial or

other high-end database features

Gatekeeper Systems ShakeCast Features and Architecture– June 2003 Slide 33

Questions and Answers

????