OptIPuter Goal:Removing Bandwidth Barriers to e-Science

ATLAS

Sloan Digital Sky Survey

LHC

ALMA

Why Optical NetworksWill Become the 21st Century Driver

Scientific American, January 2001

Number of Years0 1 2 3 4 5

Pe

rfo

rma

nc

e p

er

Do

llar

Sp

en

t

Data Storage(bits per square inch)

(Doubling time 12 Months)

Optical Fiber(bits per second)

(Doubling time 9 Months)

Silicon Computer Chips(Number of Transistors)

(Doubling time 18 Months)

The OptIPuter Project – Removing Bandwidth as an Obstacle In Data Intensive Sciences

• NSF Large Information Technology Research Proposal– UCSD and UIC Lead Campuses—Larry Smarr PI– USC, UCI, SDSU, NW, TA&M Partnering Campuses

• Industrial Partners: IBM, Sun, Telcordia/SAIC, Chiaro, Calient• $13.5 Million Over Five Years• Optical IP Streams From Lab Clusters to Large Data Objects NIH Biomedical Informatics Research Network

NSF EarthScope

http://ncmir.ucsd.edu/gallery.html

siovizcenter.ucsd.edu/library/gallery/shoot1/index.shtml

Application Barrier One:Gigabyte Data Objects Need Interactive Visualization

• Montages--Hundred-Million Pixel 2-D Images– Microscopy or Telescopes

– Remote Sensing

• GigaZone 3-D Objects– Seismic or Medical Imaging

– Supercomputer Simulations

• Interactive Analysis and Visualization of Such High Resolution Data Objects Requires: – Scalable Visualization Displays– Montage and Volumetric Visualization Software

– UIC EVL’s JuxtaView and Vol-a-Tile

OptIPuter Project Goal:Scaling to 100 Million Pixels

• JuxtaView (UIC EVL) on PerspecTile LCD Wall– Digital Montage

Viewer – 8000x3600 Pixel

Resolution~30M Pixels

• Display Is Powered By – 16 PCs with

Graphics Cards– 2 Gigabit Networking

per PC

Source: Jason Leigh, EVL, UIC; USGS EROS

NCMIR – Brain Microscopy

(2800x4000 24 layers)

Application Barrier Two:Campus Grid Infrastructure is Inadequate

• Campus Infrastructure is Designed for Web Objects– Being Swamped by Sharing of Digital Multimedia Objects– Little Strategic Thinking About Needs of Data Researchers

• Challenge of Matching Storage to Bandwidth– Need To Ingest And Feed Data At Multi-Gbps– Scaling to Enormous Capacity – Use Standards-Based Commodity Clusters (Rocks)

• OptIPuter Aims at Prototyping a National Architecture– Federated National and Global Data Repositories– Lambdas on Demand– Campus Laboratories Using Clusters with TeraBuckets– Campus Eventually with a Shared PetaCache

OptIPuter 2004 @ UCSDCoupling Linux Clusters with High Resolution Visualization

10

1

4

102

ChiaroEnstara

op-nodes-ucsd-y1.5 9/26/03 -grh

SDSCJSOESDSC

Annex

CSE 2

Fiber toCRCA

Fiber to6th College

44Bonded

GigE

1

Dell 5224

8-node cluster(shared)

8-node cluster(shared)

PreussSchool

DellGeowall

IBM 9-nodeViz Cluster

SIO SOM

Dell 5224

Sun 32-nodecomputecluster

Sun 32-nodecomputecluster

IBM 48-nodeStorageCluster21TB

Dell 5224

IBM 128-nodeCompute Cluster

(shared)

100-node cluster(shared)

Dell 5224

1

Sun 32-nodeStoragecluster

Sun 128-nodecompute(shared)

8-node cluster(shared)

2

4-nodecontrol

Sun 32-nodecomputecluster

GigE Switch10GigE Uplink

GigE Switch10GigE Uplink

UCSD 6509Shared VLAN

OptIPuter is Studying the Best Application Usagefor Both Routed vs. Switched Lambdas

• OptIPuter Evaluating Both:– Routers

– Chiaro, Juniper, Cisco, Force10

– Optical Switches– Calient, Glimmerglass

– Lightpath Accelerators– BigBandWidth

• UCSD Focusing on Routing Initially• UIC Focusing on Switching Initially• Next Year Merge into Mixed Optical Fabric

ChiaroEstara

Glimmerglass

Application Barrier Three:Shared Internet Makes Interactive Gigabyte Impossible

• NASA Earth Observation System– Over 100,000 Users Pull Data from Federated Repositories

– Two Million Data Products Delivered per Year

– 10-50 Mbps (May 2003) Throughput to Campuses– Typically Over Abilene From Goddard, Langley, or EROS

• Biomedical Informatics Research Network (BIRN) – Between UCSD and Boston– Similar Story– Lots of Specialized Networking Tuning Used

– 50-80 Mbps

• Remote Interactive Megabyte is Possible • But Interactive Gigabyte is Impossible

IP over Lambdas with Alternate Protocols

Multi-Latency OptIPuter LaboratoryNational-Scale Experimental Network

Source: Tom West, CEO NLR (Booth 3409)

Chicago OptIPuter

StarLightNU, UIC

SoCalOptIPuter

USC, UCI UCSD, SDSU

2000 Miles 10 ms

=1000x Campus Latency

“National Lambda Rail” PartnershipServes Very High-End Experimental and Research Applications

4 x 10GB Wavelengths Initially Capable of 40 x 10Gb wavelengths at Buildout

An International-Scale OptIPuter is Operational over the First Set of 76 International GE TransLight Lambdas

European lambdas to US–8 GEs Amsterdam— Chicago–8 GEs London—ChicagoCanadian lambdas to US–8 GEs Chicago—Canada —NYC–8 GEs Chicago—Canada —SeattleUS lambdas to Europe–4 GEs Chicago—Amsterdam–3 GEs Chicago—CERNEuropean lambdas–8 GEs Amsterdam—CERN –2 GEs Prague—Amsterdam–2 GEs Stockholm—Amsterdam–8 GEs London—AmsterdamTransPAC lambda–1 GE Chicago—TokyoIEEAF lambdas (blue)–8 GEs NYC—Amsterdam–8 GEs Seattle—Tokyo

UKLight

CERN

NorthernLight

Source: Tom DeFanti, EVL, UIC