The OptIPuter –From SuperComputers to SuperNetworks
GEON MeetingSan Diego Supercomputer Center, UCSD
La Jolla, CANovember 19, 2002
Dr. Larry Smarr
Director, California Institute for Telecommunications and Information Technologies
Professor, Dept. of Computer Science and Engineering
Jacobs School of Engineering, UCSD
California Has Initiated Four New Institutes for Science and Innovation
UCSBUCLA
California NanoSystems Institute
UCSF UCB
California Institute for Bioengineering, Biotechnology,
and Quantitative Biomedical Research
UCI
UCSD
California Institute for Telecommunications and Information Technology
Center for Information Technology Research
in the Interest of Society
UCSC
UCDUCM
www.ucop.edu/california-institutes
Cal-(IT)2
An Integrated Approach to the Future Internet
www.calit2.net
220 UC San Diego & UC Irvine FacultyWorking in Multidisciplinary Teams
With Students, Industry, and the Community
The State’s $100 M Creates Unique Buildings, Equipment, and Laboratories
SDSC and Cal-(IT)2 Have The Same Knowledge and Data Team by Construction!
SDSC Data and Knowledge Systems ProgramCal-(IT)2 Knowledge and Data Engineering Laboratory
Storage hardware
Networked Storage (SAN)
Grid StorageFilesystems, Database Systems
Data Mining, Simulation Modeling, Analysis, Data Fusion
Applications: Medical informatics,Biosciences, Ecoinformatics,…
Knowledge-Based Integration Advanced Query Processing
Visualization
High speed networking
sensornets instruments
The Move to Data-Intensive Science & Engineering-e-Science Community Resources
ATLAS
Sloan Digital Sky Survey
LHC
ALMA
Why Optical Networks Are Emerging as the 21st Century Driver for the Grid
Scientific American, January 2001
The Rapid Increase in Bandwidth is Driven by Parallel Lambdas On Single Optical Fibers
fc *
(WDM)
Parallel Lambdas Will Drive This DecadeThe Way Parallel Processors Drove the 1990s
CONTROL
PLANE
Clusters
DynamicallyAllocatedLightpaths
Switch Fabrics
PhysicalMonitoring
Apps Middleware
A LambdaGrid Will Be the Backbone for an e-Science Network
Source: Joe Mambretti, NU
The Next S-Curves of NetworkingExponential Technology Growth
0%
100%
Research
Experimental/Early Adopters
Production/Mass Market
Time
Technology S-Curve
Gigabit Testbeds
Connections Program
Internet2 Abilene
DWDM
Experimental Networks
Lambda Grids
~1990s 2000 2010
Networking Technology S-Curves
Tec
hn
olog
y P
enet
rati
on
Data Intensive Scientific Applications Require Experimental Optical Networks
• Large Data Challenges in Neuro and Earth Sciences– Each Data Object is 3D and Gigabytes– Data are Generated and Stored in Distributed Archives– Research is Carried Out on Federated Repository
• Requirements– Computing Requirements PC Clusters– Communications Dedicated Lambdas Over Fiber– Data Large Peer-to-Peer Lambda Attached Storage – Visualization Collaborative Volume Algorithms
• Response– OptIPuter Research Project
UIUC/NCSA
Starlight(NU-Chicago)Argonne
UChicagoIIT
UIC
Illinois Century NetworkJames R. Thompson CtrCity HallState of IL Bldg
4 pair
12 pair
4 pair
2 pair 2 pair
4 pair
18 pair
4 10 pair
12 pair
2 pair
Level(3)111 N. Canal
McLeodUSA151/155 N. MichiganDoral Plaza
Qwest455 N. Cityfront
UC Gleacher450 N. Cityfront
Illinois’ I-WIREThe First State Dark Fiber Experimental Network
Source: Charlie Catlett 12/2001
CENIC Plans to Create a Dark FiberExperimental and Research Network
Figure 2: SoCal Optical Research and Experimental Network
The SoCal Component
From Telephone Conference Calls to Access Grid International Video Meetings
Access Grid Lead-ArgonneNSF STARTAP Lead-UIC’s Elec. Vis. Lab
Can We Modify This TechnologyTo Create a GEON Research Virtual Laboratory?
• Fifteen Countries/Locations Proposing 28 Demonstrations: Canada, CERN, France, Germany, Greece, Italy, Japan, The Netherlands, Singapore, Spain, Sweden, Taiwan, United Kingdom, United States
• Applications Demonstrated: Art, Bioinformatics, Chemistry, Cosmology, Cultural Heritage, Education, High-Definition Media Streaming, Manufacturing, Medicine, Neuroscience, Physics, Tele-science
• Grid Technologies: Grid Middleware, Data Management/ Replication Grids, Visualization Grids, Computational Grids, Access Grids, Grid Portal
iGrid 2002 September 24-26, 2002, Amsterdam, The Netherlands
www.startap.net/igrid2002UIC
Sponsors: HP, IBM, Cisco, Philips, Level (3), Glimmerglass, etc.
iGrid 2002 Was Sustaining 1-3 Gigabits/s
Total Available Bandwidth Between Chicago and Amsterdam
Was 30 Gigabit/s
The NSF TeraGridA LambdaGrid of Linux SuperClusters
NCSA8 TF
4 TB Memory240 TB disk
Caltech0.5 TF
0.4 TB Memory86 TB disk
Argonne1 TF
0.25 TB Memory25 TB disk
TeraGrid Backbone (40 Gbps)
SDSC4.1 TF
2 TB Memory250 TB disk
This will Become the National Backbone to Support Multiple Large Scale Science and Engineering Projects
DataCompute
VisualizationApplicationsIntel, IBM, QwestMyricom, Sun, Oracle
$53Million from NSF
From SuperComputers to SuperNetworks--Changing the Grid Design Point
• The TeraGrid is Optimized for Computing– 1024 IA-64 Nodes Linux Cluster– Assume 1 GigE per Node = 1 Terabit/s I/O– Grid Optical Connection 4x10Gig Lambdas = 40 Gigabit/s– Optical Connections are Only 4% Bisection Bandwidth
• The OptIPuter is Optimized for Bandwidth– 32 IA-64 Node Linux Cluster– Assume 1 GigE per Processor = 32 gigabit/s I/O– Grid Optical Connection 4x10GigE = 40 Gigabit/s– Optical Connections are Over 100% Bisection Bandwidth
The OptIPuter is an Experimental Network Research Project
• Driven by Large Neuroscience and Earth Science Data– EarthScope and BIRN
• Multiple Lambdas Linking Clusters and Storage– LambdaGrid Software Stack– Integration with PC Clusters– Interactive Collaborative Volume Visualization– Lambda Peer to Peer Storage With Optimized Storewidth– Enhance Security Mechanisms– Rethink TCP/IP Protocols
• NSF Large Information Technology Research Proposal– UCSD and UIC Lead Campuses—Larry Smarr PI– USC, UCI, SDSU, NW Partnering Campuses– Industrial Partners: IBM, Telcordia/SAIC, Chiaro Networks,
CENIC• $13.5 Million Over Five Years
Metro Optically Linked Visualization Wallswith Industrial Partners Set Stage for Federal Grant
• Driven by SensorNets Data– Real Time Seismic– Environmental Monitoring – Distributed Collaboration– Emergency Response
• Linked UCSD and SDSU– Dedication March 4, 2002
Linking Control Rooms
Cox, Panoram,SAIC, SGI, IBM,
TeraBurst NetworksSD Telecom Council
UCSD SDSU44 Miles of Cox Fiber
OptIPuter NSF Proposal Partnered with National Experts and Infrastructure
Vancouver
Seattle
Portland
San Francisco
Los Angeles
San Diego(SDSC)
NCSA
SURFnet CERNCA*net4
AsiaPacific
AsiaPacific
AMPATH
PSC
Atlanta
CA*net4
Source: Tom DeFanti and Maxine Brown, UIC
NYC
TeraGrid DTFnet
CENIC
Pacific LightRail
Chicago
UICNU
USC
UCSD, SDSUUCI
switch switch
switchswitch
• Cluster – Disk
• Disk – Disk
• Viz – Disk
• DB – Cluster
• Cluster – Cluster
Medical Imaging and Microscopy
Chemistry, Engineering, Arts
San Diego Supercomputer Center
Scripps Institution of Oceanography
ChiaroEnstara
OptIPuter LambdaGridEnabled by Chiaro Networking Router
www.calit2.net/news/2002/11-18-chiaro.html
½ Mile
The UCSD OptIPuter Deployment
SIO
SDSC
CRCA
Phys. Sci -Keck
SOM
JSOE Preuss
6th College
Phase I, Fall 02
Phase II, 2003
SDSCAnnex
To Other OptIPuter Sites
Collocation point
Node M
The UCSD OptIPuter LambdaGrid Testbed
Earth Sciences
SDSC
Arts
Chemistry
Medicine
Engineering
High School
UndergradCollege
Phase I, Fall 02
Phase II, 2003
SDSCAnnex
To Other OptIPuter Sites
Collocation point
Collocation
Fast polygon and volume rendering with stereographics
GeoWall
Earth Science
GeoFusion GeoMatrix Toolkit
Underground Earth Science
Rob Mellors and Eric Frost, SDSUSDSC Volume Explorer
Dave Nadeau, SDSC, BIRNSDSC Volume Explorer
NeuroscienceAnatomy
Visible Human ProjectNLM, Brooks AFB,
SDSC Volume Explorer
3D APPLICATIONS:
+
=
OptIPuter Transforms Individual Laboratory Visualization, Computation, & Analysis Facilities
The Preuss School UCSD OptIPuter Facility
