"The OptIPuter: Enabling Campus, State, National, and Planetary-Scale LambdaGrids"

LONI Forum

Baton Rouge, LA

September 2, 2004

Dr. Larry Smarr

Director, California Institute for Telecommunications and Information Technologies

Harry E. Gruber Professor,

Dept. of Computer Science and Engineering

Jacobs School of Engineering, UCSD

OptIPuter Application Motivation: Interactive Analysis of Remote Multi-Gigabyte Data Objects

• Hundreds Of Megapixels 2-D Images– Satellite Imaging and Remote Sensing– Telescopes or Microscopy

• GigaZone 3-D (1k x 1k x 1k) Objects– Supercomputer Simulations– Seismic or Medical Imaging

• Goal: Remote Interactive Analysis and Visualization – Virtually Impossible Today– New Technology of Dedicated Lightpaths

– Bring These Capabilities to Laboratory Researchers

Cosmic Simulator with a Billion Zone and Gigaparticle Resolution

• 5123 AMR or 10243 Unigrid • 8-64 Times Mass Resolution• Can Simulate First Galaxies• One Gigazone Run:

– Output ~10 TeraByte – “Snapshot” is 100 GB– Must Visually Analyze

Compare with Sloan Survey

Source: Mike Norman, UCSD

SDSC Blue Horizon (2004)

10243 Unigrid

OptIPuter Driver: On-Line Microscopes CreatingVery Large Biological Montage Images

• 2-Photon Laser Confocal Microscope– High Speed On-line

Capability

• Montage Image Sizes Exceed 16x Highest Resolution Monitors– ~150 Million Pixels!

• Use Graphics Cluster with Multiple GigEs to Drive Tiled Displays

Source: David Lee, NCMIR, UCSD

IBM 9M Pixels

Earth System Enterprise-Data Lives in Distributed Active Archive Centers (DAAC)

SEDAC (0.1 TB)Human Interactions in

Global Change

GES DAAC-GSFC (1334 TB)

Upper AtmosphereAtmospheric Dynamics, Ocean

Color, Global Biosphere, Hydrology, Radiance Data

ASDC-LaRC (340 TB)Radiation Budget,CloudsAerosols, Tropospheric

Chemistry

ORNL (1 TB)Biogeochemical

DynamicsEOS Land Validation

NSIDC (67 TB)Cryosphere

Polar Processes

LPDAAC-EDC (1143 TB)Land Processes

& Features

PODAAC-JPL (6 TB)Ocean Circulation

Air-Sea Interactions

ASF (256 TB)SAR Products

Sea IcePolar Processes

GHRC (4TB)Global

Hydrology

EOS Aura Satellite Has Been LaunchedChallenge is How to Evolve to New Technologies

However, Average Throughput of NASA Data Products to End User is Only 50-100 Megabits/s

Tested from GSFC-ICESATAugust 2004

http://ensight.eos.nasa.gov/Missions/icesat/index.shtml

Landsat7 Imagery100 Foot Resolution

Draped on elevation data

High Resolution Aerial Photography Generates Images With 10,000 Times More Data than Landsat7

Shane DeGross, Telesis

USGSNew USGS Aerial ImageryAt 1-Foot Resolution

~350,000x350,000 Pixel Images of 350 US Cities 100 Billion Pixel Images!

States are Acquiring Their Own Dark Fiber Networks -- Illinois’s I-WIRE and Indiana’s I-LIGHT

Source: Charlie Catlett, ANL

Today Nearly Two Dozen States are

Creating Dark Fiber Networks—Including the Most Advanced

LONI!

Indiana

Regional Network Evolution

• The Fundamental Nature of Regional Networking is Rapidly Changing – The GigaPoP Model Based on Provisioned, High-

Capacity Services Steadily Is Being Replaced – On The Metro and Regional Scales

• A Model of Facility-Based Networking Built with Owned Assets has Emerged– Regional Optical Networks (RONs) – Notably, this Change Increases the Importance of

Regional Networks in the Traditional Three-Level Hierarchy of U.S. R&E Advanced Networking

Source: Steve Corbató, Internet2

Leading & Emerging Regional Optical Networks

• California (CALREN)• Colorado (FRGP/BRAN)• Connecticut (Conn. Education

Network)• Florida (Florida LambdaRail)• Georgia (Southern Light Rail)• Indiana (I-LIGHT)• Illinois (I-WIRE)• Louisiana (LONI)• Maryland, D.C. & northern

Virginia (MAX)• Michigan (MiLR)• Minnesota

• New York + New England Region (NEREN)

• North Carolina (NC LambdaRail)

• Ohio (Third Frontier Network)• Oklahoma (OneNet)• Oregon• Pacific Northwest (Lariat –

Supported by NIH)• Rhode Island (OSHEAN)• SURA Crossroads

Southeastern U.S.)• Texas (LEARN)• Utah• Virginia (MATP)• Wisconsin (WEROC)

Source: Steve Corbató, Internet2

NLR Will Provide an Experimental Network Infrastructure for U.S. Scientists & Researchers

Starting Fall 2004

“National LambdaRail” PartnershipServes Very High-End Experimental and Research Applications

4 x 10Gb Wavelengths Initially Capable of 40 x 10Gb wavelengths at Buildout

Dark Fiber: Gauging Community-Wide Progress

• Aggregate Dark Fiber Assets Acquired by/for U.S. R&E Optical Initiatives– CENIC (for CalREN & NLR) 6,200 (Route-miles)– FiberCo (via Level 3 for NLR & RONs) 5,650– SURA (via AT&T) 6,000

– Plus 2,000 Route-Miles for Research– NLR Phase 2 (WilTel & Qwest) 4,000– OARnet 1,600– ORNL (via Qwest) 900– Other Projects (IN,IL,MI,OR, …) 1,500+

• Total (conservative estimate) 25,850+– Over 55% of these Assets are in Regionals– Remainder held by NLR (11,250 Route-Miles)

Source: Steve Corbató, Internet2

Global Lambda Integrated Facility (GLIF)Creates Metacomputers on the Scale of Planet Earth

DWDM SURFnet

10 Gbit/s

SURFnet10 Gbit/s

SURFnet10 Gbit/s

IEEAF10 Gbit/s

DwingelooASTRON/JIVE

DwingelooASTRON/JIVE

PragueCzechLight

PragueCzechLight

2.5 Gbit/s

NSF10 Gbit/s

StockholmNorthernLight

StockholmNorthernLight

CA*net4

2.5 Gbit/s

New YorkMANLANNew YorkMANLAN

TokyoWIDETokyoWIDE

10 Gbit/s

10 Gbit/s

10 Gbit/s

10 Gbit/sIEEAF

10 Gbit/s

10 Gbit/s

10 Gbit/s

2.5 Gbit/s

2.5 Gbit/s

TokyoAPANTokyoAPAN

AmsterdamNetherLightAmsterdamNetherLight

GenevaCERN

GenevaCERN

LondonUKLightLondonUKLight

ChicagoStarLightChicagoStarLight

Source: Kees Neggers, SURFnet

www.glif.isCreated in Reykjavik, Iceland 2003

The OptIPuter Project – Removing Bandwidth as an Obstacle In Data Intensive Sciences

• NSF Large Information Technology Research Proposal– Cal-(IT)2 and UIC Lead Campuses—Larry Smarr PI– USC, SDSU, NW, Texas A&M, Univ. Amsterdam Partnering Campuses

• Industrial Partners– IBM, Sun, Telcordia/SAIC, Chiaro Networks, Calient, Glimmerglass

• $13.5 Million Over Five Years• Optical IP Streams From Lab Clusters to Large Data Objects NIH Biomedical Informatics NSF EarthScope

and ORION

http://ncmir.ucsd.edu/gallery.html

siovizcenter.ucsd.edu/library/gallery/shoot1/index.shtml

Research Network

OptIPuter Middleware Architecture for Distributed Virtual Computers

Layer 4: XCPNode Operating Systems

-configuration, Net Management

Grid and Web Middleware – (Globus/OGSA/WebServices/J2EE)

Physical Resources

DVC #1

OptIPuter Applications

DVC #2 DVC #3

Layer 5: SABUL, RBUDP, Fast, GTP

Real-Time Objects

Security Models

Data Services:DWTP

Higher Level Grid Services

VisualizationDVC/

Middleware

High-Speed Transport

Optical Signaling/Mgmt

Source: Andrew Chien, UCSDOptIPuter Software Systems Architect

From Grids to LambdaGrids

Creating a Model for a Campus LambdaGrid

4 1

OptIPuter UCSD / San Diego Network and Nodes -- 2004

2

ChiaroEnstara

SDSC JSOE

CSE

4 4

Bonded GigE 1

8-nodecluster

(shared)

IBM 9-node Viz cluster

SIO

SOM

IBM 48-node storage cluster

IBM 128-node CPU cluster

Sun 128-node computecluster

Sun 24-32-node

compute cluster

CRCA

6th College

3-node Viz cluster

1

IBM 9 mpixeldisplay pairs

Geowall 2Tiled Display

10

Sun 22-node

Viz cluster

10

Extreme 400Extreme 400

1

1To UCI and ISI via

CalREN-HPR

DellGeowall

Preuss

IBM 9 mpixeldisplay pairs

Dell Viz

Dell 5224

Dell 5224

Dell 5224

Dell 5224

Dell 5224

Dell 5224

Juniper T320 Cisco 6509

HP28-nodecluster

(shared)

HP4-nodecontrol

Sun 24-32-node

computecluster

Infiniband4 nodes

Infiniband64 nodes

Sun 5-node cluster

Sun24-32-node

cluster

9-nodecluster

(shared)

Extreme 400

7-node cluster(shared)

OMNInetThe Metro Area OOO Tesbed

NTONNTONCC

DWDMRAM

10 Gb Lambdas

EVL

10GE OptIPuter CAVEWAVEWill Help Launch the National LambdaRail

Next Step: Coupling NASA Centers

to NSF OptIPuter

Source: Tom DeFanti, OptIPuter co-PI

OptIPuter Scalable Visualization Systems:End User Clusters Connected With Gigabit Flows

3 Megapixels

20 Megapixels

Earth Sciences are an OptIPuter DriverMajor NASA Goddard Visit This Week

Linux Cluster with Nvidia Graphics Cards

High Bandwidth Optical Fibers Will Enable High Definition Global Virtual Teaming

In 2005 Cal-(IT)2 will Link Its Two Buildings Creating a Collaboration Laboratory

UC Irvine

UC San Diego

Ultra-Resolution OptIPuter Displays Utilizing Photonic Multicasting --Scaling to 100 Million Pixels

GlimmerglassSwitch Used to

Multicast and Direct TeraVision Stream

from One Tile to Another on the

Geowall-2

Glimmerglass Switch

Driven by

Linux Graphics Clusters

30 Megapixel High-Resolution Visualizations= ~1 Gigapixel at 30fps = ~30Gb/s Bandwidth

UIC-EVL

Currently Developing OptIPuter Software to Coherently Drive 100 MegaPixel Displays

• 55-Panel Display – 100 Megapixel

– 11x5 21” LCDs

• Developed at EVL

• Driven by 30 Dual-Opterons (64-bit)

• 60 TB Disk

• 30 10GE interfaces

• Linked to OptIPuter

• Complementary to NASA ARC HyperWall

Special issue of Communications of the ACM (CACM):Blueprint for the Future of High-Performance Networking

• Introduction– Maxine Brown (guest editor)

• TransLight: A Global-scale LambdaGrid for e-Science– Tom DeFanti, Cees de Laat, Joe Mambretti,

Kees Neggers, Bill St. Arnaud

• Transport Protocols for High Performance– Aaron Falk, Ted Faber, Joseph Bannister,

Andrew Chien, Bob Grossman, Jason Leigh

• Data Integration in a Bandwidth-Rich World– Ian Foster, Robert Grossman

• The OptIPuter– Larry Smarr, Andrew Chien, Tom DeFanti,

Jason Leigh, Philip Papadopoulos

• Data-Intensive e-Science Frontier Research– Harvey Newman, Mark Ellisman, John Orcutt

www.acm.org/cacm