Brookhaven Science AssociatesU.S. Department of Energy 1

Network Services

LHC OPN Networking at BNLSummer 2006 Internet 2 Joint Techs

John BigrowJuly 18, 2006

Brookhaven Science AssociatesU.S. Department of Energy 2

Network Services LHC Overview (very simple overview, I’m not a physicist)

• LHC / Atlas Experiments Overview (The What)

• The Physics Architecture (The Why)

• Preliminary Network and Security Architecture (The How)

Brookhaven Science AssociatesU.S. Department of Energy 3

Network ServicesCERN Accelerator Ring Aerial View

Brookhaven Science AssociatesU.S. Department of Energy 4

Network Services

Brookhaven Science AssociatesU.S. Department of Energy 5

Tier 1Tier 1

Tier2 Center

Online System

CERN ~5M SI2K >1 PB Disk Tape Robot

BBNL: NL: ~2M~2M SI2K; SI2K; 22PB PB

TapeTape Robot RobotIN2P3 Center INFN Center RAL Center

InstituteInstituteInstituteInstitute

Workstations

< GBytes/sec

2.5 Gbps

100 - 1000

Mbits/secPhysics data cache

~PByte/sec

~10 Gbits/sec

Tier2 CenterTier2 CenterTier2 Center~2.5 Gbps

Tier 0 +1Tier 0 +1

Tier 3Tier 3

Tier 4Tier 4

Tier2 Center

ATLAS Experimen

t

CERN:Outside Resource Ratio ~1:2Tier0:( Tier1):( Tier2) ~1:1:1

Tier Tier 22

Tier 0: DAQ, reconstruction, archiveTier 0: DAQ, reconstruction, archiveTier 1: Reconstruction, Tier 1: Reconstruction, simulation, simulation, archive, mining and archive, mining and (large scale)(large scale) analysis analysisTier 2+: Analysis, simulationTier 2+: Analysis, simulationTier 3+: Interactive analysisTier 3+: Interactive analysis

Network Services

Brookhaven Science AssociatesU.S. Department of Energy 6

130.199.48.0130.199.48.0

……

……

130.199.185.0130.199.185.0

130.199.48.0130.199.48.0

The same host name for dual NIC dCache door is resolved to different IP addresses depending on which DNS is inquired.

Network Services

Brookhaven Science AssociatesU.S. Department of Energy 7

Year 2004 2005 2006 2007 2008 2009 2010Remote Site(s)Tier 0 (CERN) 52 105 349 874 1,747 1,747 3,494 Tier 1's (~2 Peer sites) 37 75 250 624 1,248 1,248 2,496 Tier 2's (5 USA satellite sites) 64 128 428 1,069 2,139 2,139 4,278 Tier 3-4 (150 Individual users) 95 190 632 1,581 3,161 3,161 6,322 Total 249 498 1,659 4,148 8,295 8,295 16,590

US ATLAS Tier 1 WAN Bandwidth Requirement Estimate(Mbits/sec)

Year 2004 2005 2006 2007 2008 2009 2010US ATLAS Tier 1 Req. 249 498 1,244 4,148 8,295 9,954 16,590 RHIC Computing Facility Req. 200 500 1,023 1,286 1,847 2,422 3,381

TOTAL 449 998 2,267 5,433 10,142 12,377 19,971 BNL HEP/NP Requirement OC12 OC48 OC48 OC192 2 X _

2 x _ 3 x _

(Mbits/sec)BNL HEP/NP WAN Bandwidth Requirement Estimate

Network Services

Brookhaven Science AssociatesU.S. Department of Energy 8

Network Services

Brookhaven Science AssociatesU.S. Department of Energy 9

Other connections

Other connections

MA

N L

AN

CER

N (?

)N

LRES

net

GEA

NT,

etc

.

BN

L internal

Network Services

Brookhaven Science AssociatesU.S. Department of Energy 10

Network ServicesBNL LHC OPN Conceptual Block Diagram

LHC OPN PrivateCore Intranetwork

BNL Border RouterOptional DedicatedLHC OPN FWSMs

LHC OPN T0-T1Lambda Layer 2

Tunnel

BNL LHC OPNPrimary Distribution

Switchs

ES Net / GeneralInternet / Tier 2

BNL Internet /Tier 2 Lambda

ES Net ProvisionedCIDR IP Space

latigidlatigid latigid

BNL LHC OPN Disk Cache / Storage / Analysis FacilitiesMulti-homed

Other Tier 1 Sites

BNL Campus Network

20 Gb/Sec

20 Gb/Sec

1 Gb/Sec1 Gb/Sec 1 Gb/Sec

ACL

CIDR RestrictedDistribute ListES Net Only

ACL

ACL

NYSERNET /Broadwing

latigid

Future 10 Gb/SecUpgrades

ACL

Brookhaven Science AssociatesU.S. Department of Energy 11

Network Security Limitations

• Current firewall Architecture– 6 virtual 1 Gb/Sec EtherChannel to Catalyst backplane– Rated total throughput of 5 Gb/Sec – EtherChannel Overhead Loss – Single 1 Gb/Sec flow / interface

• New Cisco ACE blade might address these limitations

Network Services

Brookhaven Science AssociatesU.S. Department of Energy 12

Network Security Limitations (Continued)

• Current Router Architecture

– Single Access Control List (ACL) / interface- 1 inbound and 1 outbound per interface- Default behavior Implicit deny- Policy route map for traffic flow

– A single ACL can become unwieldy in a complex WAN environment (what are the network prefixes, DHCP, NAT)

– Manual changes to the route map for additional access

Network Services

Brookhaven Science AssociatesU.S. Department of Energy 13

BNL LHC Overview cont.

• Networking resources

– IP Address space allocations / access

– 10Gig interfaces / 20Gig Etherchannels

– Performance Monitoring

Network Services

Brookhaven Science AssociatesU.S. Department of Energy 14

IP Address Allocation Tier 0 to Tier 1 (BNL - CERN)

• Requires routable IP Address space

• Direct dedicated access with CERN to / from BNL

• Limited route advertisements between T0 and T1– For the LHC OPN Circuit BNL will use 192.12.15.0/24– No direct T1 to T1 access through CERN at this time

Network Services

Brookhaven Science AssociatesU.S. Department of Energy 15

BNL OPN to Tier 2 and others

• Tier 2 and other traffic dependant on Internet connectivity

– Path to BNL via all service providers (ES Net now, NYSERNET, Broadwing in the future ?)

– Dedicated paths to other institutions welcome (you buy)

Network Services

Brookhaven Science AssociatesU.S. Department of Energy 16

Network Services

Amon Mutt

TefnutShu

Anubis

Nephthys

Isis

Osiris

CoreSW9

SW7

Direct Layer 2 Interface to CERNT0 - T1

1 x 10G3 Peerings

Internet Peer with ES Net

1 x 10G

PreliminaryBNL 10 /20 Gig-E LHC OPN

Initial Architecture

BNL LHC OPN

Gateway ACL

Brookhaven Science AssociatesU.S. Department of Energy 17

Future BNL LHC OPN Enhancements

• Dedicated Cisco Firewall Service Modules (ACE) when available

– Eliminate router ACL Functionality / Maintenance– Connection Logging– Each FWSM circuit will not impede the 10 Gb/Sec. – Stateful FWSM redundancy

• IDS / IPS when available

Network Services

Brookhaven Science AssociatesU.S. Department of Energy 18

Network Services

Amon Mutt

TefnutShu

Anubis

Nephthys

Isis

Osiris

CoreSW9

SW7

Direct Layer 2 Interface to CERNT0 - T1

1 x 10G

Internet Peer with ES Net

1 x 10G

BNL Campus NetworkIncluding Near-Term Upgrades

StatefulLink

NYSERNETBroadwing

FE

FailoverFailover

FWSM

Building AccessLayer Switch (Typical

Deployment)

DL1 DL2BNL LHC OPN

Brookhaven Science AssociatesU.S. Department of Energy 19

Network Services Mon

• browser-based IP service monitor• Internet-centric WAN based monitor application• Interrogates essential BNL network services

Brookhaven Science AssociatesU.S. Department of Energy 20

Brookhaven Science AssociatesU.S. Department of Energy 21

Network Services MonaLisa

• Java based SNMP monitoring tool

External WAN based monitor

Tracks BNL 10G/Sec. Interfaces Firewall Service Module 20 Gb/Sec. Uplinks to the BNL core

Brookhaven Science AssociatesU.S. Department of Energy 22

Network Services

Brookhaven Science AssociatesU.S. Department of Energy 23

Network Services

Brookhaven Science AssociatesU.S. Department of Energy 24

Network Services Cacti

• SNMP monitoring tool• Replacement for MRTG• Tracks most BNL core network interfaces• Firewall Service Module EtherChannel interfaces also

Brookhaven Science AssociatesU.S. Department of Energy 25

Network Services

Brookhaven Science AssociatesU.S. Department of Energy 26

Network Services

Brookhaven Science AssociatesU.S. Department of Energy 27

Network Services

Brookhaven Science AssociatesU.S. Department of Energy 28

Network Services

Brookhaven Science AssociatesU.S. Department of Energy 29

Network Services

Brookhaven Science AssociatesU.S. Department of Energy 30

Thanks (a few kind words to so many)

• Thanks to the many individuals and groups who have donated their time, code, and talents to make the Internet what it is today. Without their efforts, this infrastructure we take for granted would not exist. We owe many our gratitude.

Network Services

Brookhaven Science AssociatesU.S. Department of Energy 31

Questions/Comments

???

Network Services

Brookhaven Science AssociatesU.S. Department of Energy 32

BNL Points of Contact Scott Bradley, Manager of Network Services

• 631.344.5745, bradley@bnl.gov

John Bigrow, Senior Network Architect• 631.344.2648, big@bnl.gov

Network Services