International Networks International Networks (and Networking)(and Networking)
John Silvester
Professor of Electrical Engineering, USCBoard Member, CENIC
Consultant to NLR
CENIC Annual Meeting, March 12th, 2007, La Jolla, CA
CENIC 2007
2
OutlineOutline
• Networks around the Globe– North America– GEANT2– CLARA – TEIN2– EUMEDCONNECT – APAN– (GLORIAD)
• International Networking (from the US perspective)
• Exchanges
CENIC 2007
3
North AmericaNorth America
• Abilene => New Internet2• NLR• ESNET• Canarie• NREN• DREN• RONS and other Regionals• ….
CENIC 2007
4
GÉANT2
CENIC 2007
5
CLARA
CENIC 2007
6
TEIN2
CENIC 2007
7
EUMED ConnectEUMED Connect
CENIC 2007
8
CENIC 2007
9
CENIC 2007
10
OutlineOutline
• Networks around the Globe• International Networking (from the US
perspective) – Transpac2– WHREN/LILA– Gloriad– TransLight/Starlight– TransLight/PacificWave– GLIF
• Exchanges
CENIC 2007
11
Transpac2Transpac2
CENIC 2007
12
WHREN-LILA Funded PlanWHREN-LILA Funded Plan
• (1) 1.2Gbps link evolving to 2.5Gbps + dark fiber segment
• U.S. landings in Miami and San Diego
• Latin America landing in Sao Paulo and Tijuana
• $5M over 5 years
10
CENIC 2007
13
CENIC 2007
14TransLight/StarLightTransLight/StarLight
Two Trans-Atlantic LinksTwo Trans-Atlantic Links
GÉANT2 PoP @ AMS-IENetherLight
StarLight
MAN LAN
• OC-192 routed connection between MAN LAN in New York City and the Amsterdam Internet Exchange that connects the USA Abilene and ESnet networks to the pan-European GÉANT2 network
• OC-192 switched connection between NLR and RONs at StarLight and optical connections at NetherLight; part of the GLIF LambdaGrid fabric
www.startap.net/translight
IRNC Services
MAN LAN: Layer 3 (default)
StarLight: Layer 2 VLANs, engineered upon request.
CENIC 2007
15
Pacific Connectivity and SX TransPORTPacific Connectivity and SX TransPORT
CENIC 2007
16IRNC TL/PW Pacific ConnectionsIRNC TL/PW Pacific Connections
TλEX Tokyo PW-Seattle
AARnet POP Sydney
Hawaii
OahuPW-LA
CLARA, CUDI POPs (Tijuana)
AARnet-SX Transport
IEEAF Link
CENIC 2007
17 Mauna Kea Observatories Communication Network
CSO Gemini IRTF JCMTSMA UKIRTUH-2.2UH-0.6VLBA CFHTKeck
CSO
Gemini
JAC
Subaru
SMA
CFHT
Keck
IfA Hilo
Verizon Summit
Verizon Hilo
VerizonHumuula
VerizonWaimea
U N I V E R S I T YU N I V E R S I T Y
UH Hilo
Subaru
OC12
OC12To summit
OC3to Japan
Verizon SONET
Verizon DS3 (private)
Verizon DS3 (shared)
IfA managed OC3
IfA managed 100Mb
IfA managed GE
56kb
VLBA
Summit
Hilo
Waimea
T1 framerelay
Hale Pohaku
IfA Summit
CENIC 2007
18
CUDI and CLARA ConnectionsCUDI and CLARA Connections
PWLAEXCH
ONS
SW
CW
DM SW
CW
DM
CW
DM SW
CW
DM
ONS
CUDI
CLARA
Bestel, TJGC, SD
LA (818)
SDSC
CenicOptical
backbone
CenicOptical
backbone
TLPW Project:Extension of 1G circuits to connect CUDI and CLARA
CENIC 2007
19 International Connectivity (GLIF)International Connectivity (GLIF)
www.glif.isVisualization courtesy of Bob Patterson, NCSA.
and Exchange Patchpanelsand Exchange Patchpanels
CENIC 2007
20
OutlineOutline
• Networks around the Globe• International Networking (from the US
perspective)• Exchanges
– Starlight– Manlan– PacificWave– AtlanticWave
CENIC 2007
21
UKLight
JGN II
2xOC-192to Amsterdam
IRNC and SURFnet
4xOC-192to Canada, Seattle,
Korea, Taiwan, NYC, Amsterdam, GLORIAD
OC-192 to London
36x10GE100x1GE
16-processor cluster
ESnet, NREN,NASA / GSFC NISN, DREN, USGS, etc.
GE ElectronicallySwitched
10GE ElectronicallySwitched/Routed
NortelLayer 1Switch
1
Many Clusters at StarLight, NU, NCSA, UIC
Nx10GE Nx1GE
StarLightGLIF GOLE
October 2006
10GE
TeraGrid Juniper T640
Nx10GENxOC-192
To NCSA/SDSCANL/ETF
10GE
MidWestMREN
DS-3 to Hong Kong/HARnet
10GE
OC-192 toCERN
Fermilab DWDM10GE
10GE
CalTech Juniper T320
10GENx10GE, NxGEOC-3, DS-3
2xOC-192
OC-192
10GE, Nx1GEvia MREN Force 10
4xOC-192
2xOC-192
Switch at PNGWPSwitch/Clusters/4K
at UCSD Calit2
2x10GE over NLR
OC-192 Electronically
Switched
6
SINET OC-12 to Japan
ASNetOC-48 to Taiwan
(10GE soon)
2
DS-3 to China/CERnet
OMNInet2
NxOC-192via DXs
10GE NLR
LONI, others on NLR
Switch atMcLean, Virginia
Clusters at GSFC,JCVI, others
10GE over CAVEWave/NLR
Nx10GE
Force10
HDXc
64x64GMPLSMEMSSwitch
Calient
Abilene
TransLight/StarLightTransLight/StarLight StarLight Engineering StarLight Engineering
www.startap.net/starlight/ENGINEERING/engArch.html
CENIC 2007
22TransLight/StarLightTransLight/StarLight
StarLight Connected Networks StarLight Connected Networks
MCNC/EnLIGHTenedMiLRMRENNational LambdaRailNISNOMNInetSouthern Light RailTeraGridTransLightUltraScience NetUSGSWisc Wave
National and Regional Networks
www.startap.net/starlight/NETWORKS Source: Linda Winker
ASNetCANARIECERN/LHCNETCERNET/NSFCNETCESNET (Czech Repubic −
Coming December 2006)GLORIAD-China/CSTnetGLORIAD-RussiaGLORIAD-KREONet2/KORENHARNETJGN-II SINET SURFnetTaiwanLight/TWARENTransLight/StarLight IRNCUKLight
International Advanced Networks
AbileneBOREAS (Coming
February 2007)CAVEwaveDRENESnetFermi LightPathHOPII-LightI-WIRELONI
CENIC 2007
23
15 x 10Gb
6 x 1GE
MAN LAN is an open exchange point for the research and education community; built within the fiber meet-me room (FMMR) in the carrier hotel at 32 Avenue of the Americas, New York, NY, MAN LAN is a collaborative effort of
Internet2NYSERNetThe Global NOCIEEAF
Connectors to the exchange point determine how their ports and interconnects are configured.
MAN LANMAN LAN
CENIC 2007
24
AMPATH Ethernet FabricCisco 7609, Foundry FastIron
Lucent CBX 500
Cisco ONS15454
STM-16 PortCurrently configured as VC4-8c with
1.22 Gbps effective bandwidth.
MIAMI
RNP
ANSP
CLARA
Abilene AtlanticWaveNLR
UltraLight
WHREN-LILA Cisco ONS 15454
SONET
Gigabit Ethernet
10 Gigabit Ethernet
LILA East Design:LILA East Design:AMPATH International Exchange PointAMPATH International Exchange Point
12
CENIC 2007
25LILA East Design:LILA East Design:
Sao Paulo Distributed Exchange PointSao Paulo Distributed Exchange Point
Sao PauloCisco 15454
RedCLARArouter
RNPswitch
ANSP switch
RNPswitch
RNProuter
WHREN-LILA
RedCLARA
MiamiCisco 15454
Redundant dark fiber
USP
HEPGridUERJ
SPRACEUNESP
BARUERICOTIA
SDH/SONET
N X GbE
Sao Paulo has become a distributed exchange point, interconnecting ANSP, CLARA, RNP, supporting a distributed CMS Tier-2 facility,
with inter-regional connections to U.S. R&E networks and E.U. GEANT
13
CENIC 2007
26
““Layer 3” R&E Networks and Pacific WaveLayer 3” R&E Networks and Pacific Wave
AARNET
GEMNET
GLORIAD
IEEAF
KREONET
MIMOS (Malaysia)
QATAR
SINET
SINGAREN
TANET/TWAREN
TRANSPAC2
ABILENE (Internet2)
CALREN (HPR)
DREN ESNET
NASA
NLR-Packetnet
NREN
PNWG
Pacific Wave
CA*NET4
CLARACUDI
CENIC 2007
27
PacificWave SeattlePacificWave Seattle
Seattle WestinCISCO 6509
NLR Lambda to LAX (via Sunnyvale)
Abilene
AARNet
CA*NET4
GEMnet
SingAREN
TANET2/TWARENATTBI/Comcast
DREN
KREONet2
ESnet
Microsoft
PNWGP
NLR Lambda to STARLIGHT
WIDE/TLEX(IEEAF)
NLR-Packetnet
Does not show “lightpath” connections.
1 Gigabit
10 Gigabit
KAREN
CENIC 2007
28
Pacific Wave - SunnyvalePacific Wave - Sunnyvale
NLR TO SEATTLE PWAVE
L3 SVL6509
CENIC SVL HPR
NLR TO LA PWAVE
PAIX
ESNET
Does not show “lightpath” connections
1 Gigabit
10 Gigabit
NREN
CENIC 2007
29
Pacific Wave - Los AngelesPacific Wave - Los Angeles
1 Wilshire6509
600 W 7th6509
818 W 7th6509
CalREN-HPR
10 GE NLR LambdaTo Seattle PWave(via Sunnyvale)
Abilene
TWAREN
TRANSPAC2
Los Nettos
Qatar Foundation
NII/SINETAARNet
(3)Singaren
Mimos Berhad(Malaysia)
Cinegrid
Does not show “lightpath” connections1 Gigabit
10 Gigabit
CUDI
CLARANLR Packetnet
CENIC 2007
30
Overall PW Exchange TrafficOverall PW Exchange Traffic
Live link to pw stat page
The data provided here is an aggregate of inbound and outbound IP packets on Participant ports.
Live link to current stat
CENIC 2007
31
Changing Requirements for ExchangesChanging Requirements for Exchanges
• The Exchange points need to provide a wider array of services:– Layer 3 Interconnect (routed)– Layer 3 Interconnect (over layer 2 switch)– Layer 2 Interconnect (Ethernet switching)– Layer 1 Interconnect (wave switching)– Layer 0 Interconnect (fiber interconnect)
• PacificWave is moving in this direction to allow us to continue to participate in the world of Global R&E networking
CENIC 2007
32
February 2005 (Start of IRNC)• 10G: 15 ports − UWisc, CAVEwave, UvA Itanium pc, JGN-2, Abilene, CERN, Teragrid,
NASA GSFC, EVL/IWIRE, NCDM pcs (2x10G), ESnet, Fermilab• 1GE: 42 ports − UKLight (4x1 GE), GLORIAD (2x1 GE), CERN, HEAnet, TAnet2,
KREONet2, Queen's College Canada, Renater, ASCC (2x1 GE), Calient PXC (5x1 GE), SURFnet, StarLight iperf server, SINET, EVL disk server, HARnet, Canarie (2x1 GE), EVL cluster (5x1 GE), UCSD PCs (2x1 GE), NASA PC (5x1 GE), EVL (4x1 GE)
October 2005 (added since February 2005)• 10G: 20 ports − GLIF HDXc (2x10G), NCSA OptIPuter, LSU/LONI, NASA PC, EVL disk
server, UWisc, CAVEwave, UvA Itanium PC, JGN-2, Abilene, CERN, TeraGrid, NASA GSFC, EVL/IWIRE, NCDM PCs (2x10G), ESnet, Fermilab
• 1GE: Total 58 − YottaYotta disk, Nortel Labs Canada, NISN, DREN, NREN, ESnet, GLORIAD, DOT (4x1 GE), EVL (4x1 GE), NCDM PCs (7x1 GE), StarLight NDT server, UKLight (4x1 GE), GLORIAD (2x1 GE), CERN, TAnet2, KREONet2, Queen’s College Canada, Renater, ASCC (2x1 GE), Calient PXC (5x1 GE), SURFnet, StarLight iperf server, SINET, EVL disk server, HARnet, CANARIE (2x1 GE), EVL cluster (5x1 GE), UCSD PCs (2x1 GE), NASA PC (5x1 GE)
TransLight/StarLightTransLight/StarLight StarLight Switch Ports: A Metric StarLight Switch Ports: A Metric
CENIC 2007
33TransLight/StarLightTransLight/StarLight
StarLight Switch Ports: A Metric StarLight Switch Ports: A MetricOctober 2006• 10G ports: 27 (12 more than March 2005) − Abilene, CERN, Teragrid, Fermilab, EVL
(2x10G), CAVEwave, NCSA OptIPuter, GLIF HDXc (4x10G), LSU, NASA pc, JGN-2, NLR L2, NLR L3, NASA GSFC, UIC NCDM PCs (2x10G), EVL PC, CAVEwave-DC, GaTech, TAnet2/TWAREN, TransLight/Pacific Wave, ESnet, GLORIAD
• 1GE ports: 58 (16 more than March 2005) − UKLight (5x1 GE), GLORIAD (2x1 GE), CERN, TAnet2, Nortel Labs, CERnet, KREONet2 (2x1 GE), Queen’s College Canada, Carleton Univ Canada, DREN, Renater, ASCC (2x1 GE), Calient PXC (5x1 GE), SINET, NISN, Simon Frasier Univ Canada, HARnet, USGS, HKLight, CANARIE (2x1 GE), SURFnet, DOT (4x1 GE), ESLEA/UKLight PC, EVL PCs (2x1 GE), EVL (8x1 GE), StarLight PCs (iperf/BWCTL, NDT), UCSD PCs (2x1 GE), UIC NCDM PCs (7x1GE)
• Many 10G Links come to StarLight but do not come to the StarLight E-1200 (e.g., some of NLR/TeraGrid, ESnet, CA*net 4, MREN)
• iCAIR: installing 10 node cluster: each node has a 10Gb NIC/fiber path to SL • Researchers and networks/computers come to StarLight because it’s easy
CENIC 2007
34Star Star TransLight/StarLightTransLight/StarLight
StarLight GOLE Configuration StarLight GOLE Configuration
14 StarLight GLIF (lightpath sharing) 10GE/OC-192 L2 Links
14 Additional 10GE and 58 1GE links to SL E-1200 not shown
23 Additional NLR (13 10GE, 7 OC-192 and 3 spare) not shown
12 Regional 10GE and numerous 1GE connections not shown
www.glif.is/resources/starlight-topology.jpgSource: Linda Winkler
CENIC 2007
35
““Layer 3” R&E Networks and Pacific WaveLayer 3” R&E Networks and Pacific Wave
AARNET
GEMNET
GLORIAD
IEEAF
KREONET
MIMOS (Malaysia)
QATAR
SINET
SINGAREN
TANET/TWAREN
TRANSPAC2
ABILENE (Internet2)
CALREN (HPR)
DREN ESNET
NASA
NLR-Packetnet
NREN
PNWG
Pacific Wave
CA*NET4
CLARACUDI
CENIC 2007
36
Lightpath Networks and PWLightpath Networks and PW
Others in the planning stages
Gloriad
Cinegrid
TRANSPAC (?)
AARNET (S)
(to TλEX) IEEAF
CA*NET4
TransLight SEA-CHI (CRW)
NLR Waves
NLR (Layer 2)
CALREN-XD
HOPI, Internet2 (newnet)
PNWG
Pacific Wave
CENIC 2007
37
PacificWave GOLE DiagramPacificWave GOLE Diagram
