1

ESnet Planning for theLHC T0-T1 Networking

William E. Johnston ESnet Manager and Senior Scientist

Lawrence Berkeley National Laboratory

ES

net

Scie

nce D

ata

N

etw

ork

(S

DN

) core

TWC

SNLL

YUCCA MT

BECHTEL-NV

PNNLLIGO

INEEL

LANL

SNLAAlliedSignal

PANTEX

ARM

KCP

NOAA

OSTI ORAU

SRS

JLAB

PPPLINEEL-DCORAU-DC

LLNL/LANL-DC

MIT

ANL

BNL

FNALAMES

4xLAB-DC

NR

EL

LLNL

GA

DOE-ALB

GTN&NNSA

International (high speed)10 Gb/s SDN core10G/s IP core2.5 Gb/s IP coreMAN rings (≥ 10 G/s)OC12 ATM (622 Mb/s)OC12 / GigEthernetOC3 (155 Mb/s)45 Mb/s and less

Office Of Science Sponsored (22)NNSA Sponsored (12)Joint Sponsored (3)

Other Sponsored (NSF LIGO, NOAA)Laboratory Sponsored (6)

QWESTATM

42 end user sites

ESnet IP core

SINet (Japan)Japan – Russia (BINP)CA*net4 France

GLORIAD Kreonet2MREN NetherlandsStarTap TANet2Taiwan (ASCC)

AustraliaCA*net4Taiwan (TANet2)Singaren

ESnet IP core: Packet over SONET Optical Ring and

Hubs

ELP HUB

ATL HUB

DC HUB

peering points

MAE-E

PAIX-PAEquinix, etc.

PN

WG

Po

P

SEA HUB

ESnet High-Speed Physical Connectivity toDOE Facilities and Collaborators, Summer 2005

IP core hubs SNV HUB

Abilene high-speed peering points

Abilene

Ab

ilen

e

CERN(LHCnet – partDOE funded)

GEANT - Germany, France, Italy, UK, etc

NYC HUB

Starlight

Chi NAP

CHI-SL HUB

SNV HUB

Ab

ilene

SNV SDN HUB

JGI

LBNL

SLACNERSC

SND core hubs

SDSC HUB

Equinix

MA

N L

AN

Abi

lene

MAXGPoP

SoXGPoP

SNV SDN HUB

ALB HUB

ORNL

CHI HUB

3

ESnet

• The IP core is primarily a layer 3 infrastructureo However, supports layer 2 via MPLS

o Directly connects sites

o Provides global peering for sites

• The SDN core is primarily a layer 2 infrastructureo Targeted at providing virtual circuit services

GEANT (Europe)

Asia-Pacific

New York(AOA)

Chicago (CHI)

Sunnyvale(SNV)

Washington, DC (DC)

El Paso (ELP)

Primary DOE Labs

Existing IP core hubs

ESnet Target Architecture: IP Core + Science Data Network + MANs

Possible new hubs

Atlanta (ATL)

CERN

Core loops

Seattle(SEA)

Albuquerque (ALB)

New hubs

SDN hubs

Aus.

Aus.

Production IP coreScience Data Network coreMetropolitan Area NetworksLab suppliedInternational connections

ESnetScience Data Network

(2nd Core)

ESnetIP Core

MetropolitanArea Rings

5

Site gateway routersite equip. Site gateway router

ESnet production

IP core

ANLFNAL

ESnet MAN Architecture (e.g. Chicago)R&E peerings

monitor

ESnet management and

monitoring

ESnet managedλ / circuit servicestunneled through the IP backbone

monitor

site equip.

ESnet production IP service

ESnet managedλ / circuit services

T320

International peerings

Site LAN Site LAN

ESnet SDN core

T320

2-4 x 10 Gbps channels

core router

switches managingmultiple lambdas

core router

Starlight Qwest

6

The First ESnet MAN: SF Bay AreaChicago

El Paso

Seattle and Chicago

LA andSan Diego

SF Bay Area

λ4 future

λ3 future

λ2 SDN/circuits

λ1 production IP

SLAC

Level 3hub

Qwest /ESnet hub

SNLL

Joint Genome InstituteLBNL

NERSC

LLNL

•2 λs (2 X 10 Gb/s channels) in a ring configuration, and delivered as 10 GigEther circuits

•Dual site connection (independent “east” and “west” connections) to each site

•Will be used as a 10 Gb/s production IP ring and2 X 10 Gb/s paths (for circuit services) to each site

•Qwest contract signed for two lambdas 2/2005 with options on two more

•Project completion date is 9/2005.

Qwest-ESnetnational core

ringNational Lambda

Rail circuits

ESnet MAN ring

7

FNAL ESnet site gateway router

site equip.

Qwest hub(NBC Bld.)Starlight

ESnet IP core

FNAL

Shared w/ IWire

ESnet

ESnet Near-Term Planning for FNAL

FNAL

CERN

ESnet Switch/RTR

ESnet/Qwest

ORNL OC192

ESnet fiber

T320

NRL,UltraScienceNet, etc.

T320

ESnet SDN core

All circuits are 10Gb/s

Notes

Qwest

IWire

FNAL

8

ESnet Planning for BNL (Long Island MAN Ring)

Engineering Study for LI MAN

Other connections

Other connections

Denver

Seattle

Su

nn

yv

ale

LA

San Diego

Chicago

PittsWash DC

Raleigh

Jacksonville

Atlanta

KC

Baton Rouge

El Paso - Las Cruces

Phoenix

Pensacola

Dallas

San Ant.Houston

Albuq. Tulsa

New YorkClev

Proposed ESnet Lambda InfrastructureBased on National Lambda Rail – FY08

NLR wavegear sites

NLR regeneration / OADM sites

Boise

10

DENDEN

ELPELP

ALBALBATLATL

Metropolitan Area Rings

ESnet Goal – 2007/2008

Aus.

CERN

Europe

SDGSDG

AsiaPacSEASEA

Major DOE Office of Science SitesHigh-speed cross connects with Internet2/Abilene

New ESnet hubsESnet hubs

SNVSNV

Europe

Japan

CHICHI

Science Data Network coreLab suppliedMajor international

Production IP ESnet core

DCDC

Japan

NYCNYC

Aus.

MetropolitanAreaRings

• 10 Gbps enterprise IP traffic • 40-60 Gbps circuit based transport

ESnetScience Data Network

(2nd Core – 30-50 Gbps,National Lambda Rail)

ESnet IP Core(≥10 Gbps)

10Gb/s10Gb/s30Gb/s

40Gb/s

11

OSCARS: Guaranteed Bandwidth Service

• A high-priority R&D topic in “DOE Science Networking Challenge: Roadmap to 2008”

• The MICS funded OSCARS project is implementing dynamically provisioned circuit switching in ESneto On-demand Secure Circuits and Advance Reservation

System (OSCARS)

o End-to-end provisioning will be initially provided by a combination of Ethernet switch management of λ paths in the MAN and Ethernet VLANs and/or MPLS paths in the ESnet core

o Provisioning will initially be provided by manual circuit configuration, on-demand in the future

o A technology collaboration with Internet2/Abilene

12

OSCARS: Guaranteed Bandwidth Service

• Progresso Testing OSCARS Label Switched Paths (MPLS based

virtual circuits)- A static LSP between BNL and FNAL has been configured for the

sites to test their infrastructure

- Multiple static LSPs have been configured between GA and NERSC to test local and wide-area QoS

o Collaboration – code is being jointly developed with Internet2's Bandwidth Reservation for User Work (BRUW) project

13

OSCARS: Guaranteed Bandwidth Service

usersystem2

usersystem1

site B

resourcemanager

resourcemanager

resourcemanager

polic

er

auth

oriz

atio

n

shap

er

site A

allocationmanager

polic

er

bandwidthbroker

resourcemanager