Date post: | 29-Dec-2015 |
Category: |
Documents |
Upload: | hugh-powell |
View: | 214 times |
Download: | 0 times |
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
DOE-UltraScience Net (& network infrastructure) Update
JointTechs MeetingFebruary 15, 2005
W. R. Wing
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Motivation - DOE Needs Extreme Networking 160-200 Gbs throughput by 2008
Probably only achievable by circuit bonding
0.1% packet re-order and jitter control Probably only achievable at SONET layer
Line rate, provably-secure, connections Only available at SONET or optical layer
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
The Well Known Bandwidth Problem
DOE needs 160 - 200 Gbs by 2008 Typical multi-stream throughput limited to
25+/- Gbs, and programming multi-steam is hard - hero efforts yield ~30 Gbs
Typical single-steam throughput to a single application ~1Gbs, even on a tuned network - hero effort yields 5-6+ Gbs
Networks will soon offer 40 Gbs channels Firewalls throw it away anyway
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Even Without Firewalls -We Have a Technology Problem
• DOE needs 160-200 Gbs by 2008• This is on the “blue line”
• This isn’t a problem routers are going to fix - bonded IP channels ONLY work with VERY large numbers of streams (as Teragrid has discovered)• Bonding at layer-2 (Ethernet) is just as bad• Solution is circuit switching and Layer-1 bonding
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Jitter Control
Needed to deconstruct data sets, needed for remote vis, and for remote instrument control or steering
Routers and switch-routers are path deterministic, but not time-deterministic
Only at layer-1 (or SONET) do you get time-determinism
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Security
Security requirements will get more and more onerous
Solutions involving IPSec, VPN’s, and encryption have more and more trouble running at line speed
Alternate solution is to use optical/SONET circuits with provable, known end-points and inherent immunity to injected traffic
Again - only available at layer-1
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
In Summary Only at layer-1 do you get:
Zero packet re-ordering Zero jitter Zero drops due to Congestion Known (by definition) paths and end points
SONET (or below) can do this Ethernet switches and switch/routers can’t MPLS can’t
Layer-1 circuits can: Bypass firewalls Carry non-IP frames (e.g., Fiber-Channel over SONET) Easily (transparently) support parallel, bonded circuits
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
How do we get there from here?
It requires a research network we control at least down to the SONET layer
It requires a research network with significant span
It requires a research network with at least two lambdas
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
UltraScience Net Research Testbed
Building an extended-regional lambda-switching testbed
Connect to NLR in Atlanta and Chicago Use asset-trading to extend reach (Sunnyvale and
East Coast) Provide an evolving matrix of switching
capabilities Separately fund research projects (e.g., high-
performance protocols, control, visualization) that will exercise the network and directly support applications at the host institutions
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
In More Detail -C
hic
ag
oC
hic
ag
o
Sc
hn
eid
er
Sc
hn
eid
er
Mo
no
nM
on
on
Cra
wfo
rds
vil
leC
raw
ford
sv
ille
Ind
ian
ap
oli
sIn
dia
na
po
lis
Bo
wli
ng
Gre
en
Bo
wli
ng
Gre
en
Na
sh
vil
leN
as
hv
ille
Ch
ris
tia
na
Ch
ris
tia
na
Se
ym
ou
rS
ey
mo
ur
Lo
uis
vil
leL
ou
isv
ille
Up
ton
Up
ton
Ste
ve
ns
on
Ste
ve
ns
on
Ch
att
an
oo
ga
Ch
att
an
oo
ga
Da
lto
nD
alt
on
Ac
wo
rth
Ac
wo
rth
Atl
an
taA
tla
nta
OLAOLA
KnoxvilleKnoxville
ORNLORNL
2x10GbE ORNL TeraGRID2x10GbE ORNL TeraGRID
OLA - 2OLA - 2
MemphisMemphis
OLA -1OLA -1
2xO
C19
2 –
UT
Ter
aGR
ID2x
OC
192
– U
T T
eraG
RID
2xOC192 – UT TeraGRID
2xOC192 – UT TeraGRID
2x10
Gb
E –
UT
Ter
aGR
ID2x
10G
bE
– U
T T
eraG
RID
T640T640
T320T320
T640T640
CICI
CDCD
CICI
2xOC192 – ORNL TeraGRID2xOC192 – ORNL TeraGRID 2x10GbECD bypass for future 10GbECD switching
2x10GbECD bypass for future 10GbECD switching
20 x Gig-E???20 x Gig-E???
ONLINE M
etro
???`
ONLINE M
etro
???`
ORNL TeraGRIDORNL TeraGRID
UT TeraGRIDUT TeraGRID
N x
10
Gb
E
N x
10
Gb
E
N x 10GbE
N x 10GbE
N x 10GbE, UltranetN x 10GbE, Ultranet
TN.eduTN.edu
F10F10
GbE / 10GbEGbE / 10GbE
Gb
E /
10
Gb
E
Gb
E /
10
Gb
E
Gb
E /
10
Gb
E
Gb
E /
10
Gb
E
Gb
E /
10
Gb
E
Gb
E /
10
Gb
E
GbE / 10GbEGbE / 10GbE
OC192’sOC192’s
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Connections to National Nets
ORNL
ORNL-TVA
Internet2
NLR
ORNL
ESnetTeraGrid
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Research Projects A progression of switching approaches
Study/compare SONET, MPLS(GMPLS), and all optical New all-optical technologies coming (e.g., laser tuning)
Fast Local Storage Storage Depots Transport-optimized storage
Progression of experimental point-to-point transport technologies Fiber channel Infiniband LAN-PHY Ethernet initial transport
Circuit-Switched backbone, Frame-Switched edge
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Digression: LAN-PHY vs. WAN-PHY
Conventional Wisdom: LAN-PHY will win Cheaper, faster, etc. (based on POS experience) False cost model (extra processing) Ignore SONET advantages
New Transponders are transparent Software selectable OC192 (WAN–PHY) or LAN–
PHY Costs now equal Advantage now WAN-PHY
DCC and OAM non-trivial considerations
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
PoS vs. Ethernet
DATAIP IPPOS PAD POS
DATAIP IPE EPAD
DATAIP IP
DATAIP IPPOS PAD POSSONET SONET
DATAIP IPE EPAD
DATAIP IP
DATAIP IPE EPADSONET SONET
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Status and Schedule
All contracts signed Fiber, Co-Lo space and power, equipment and
installation, smart-hands Equipment shipping, built-out started
Chicago-Sunnyvale paced by NLR cross-connects in Chicago (Level(3) to Starlight) NLR supplying 10Gig-E (LAN-PHY) initially PNNL fiber schedule will pace their connection
First Chicago-Sunnyvale traffic this month Details on following VuGraphs -
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Phase-1 (February)
Force10E600
Sunnyvale
Chicago
Force10E600
Seattle
10G WanPHY
OC-192 SONET
OC-192 SONET (integrated optics)
10G LanPHY
1G Ethernet
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Phase-2 (February-March)
VL2020CienaCD-CI
Force10E600
Sunnyvale
Chicago
Atlanta
CienaCD-CI
Force10E600
CienaCD-CI
SycamreSN16000
CienaCD-CI
SycamreSN16000
TurinTR2000
Seattle
ORNL
10G WanPHY
OC-192 SONET
OC-192 SONET (integrated optics)
10G LanPHY
1G Ethernet
N x 1Gig-E
ORNLM160
SOXM320
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Phase-3 (March - April)
VL2020CienaCD-CI
Force10E600
Sunnyvale
Chicago
Atlanta
CienaCD-CI
Force10E300
Force10E600
CienaCD-CI
SycamreSN16000
Force10E300
SycamreSN16000
TurinTR2000
Seattle
ORNL
10G WanPHY
OC-192 SONET
OC-192 SONET (integrated optics)
10G LanPHY
1G Ethernet
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Phase-4 (April - May)
Force 10E300
Sunnyvale
Chicago
Atlanta
CienaCD-CI
CienaCD-CI
CienaCD-CI
SycamreSN16000
SycamreSN16000
CienaCD-CI
Force 10E300
Force 10E300
Force 10E300
VL2020
10G WanPHY
OC-192 SONET
OC-192 SONET (integrated optics)
10G LanPHY
1G Ethernet
Seattle
PNNL
ORNL
TurinTR2000