Global Networking for the LHC

Artur Barczyk

California Institute of Technology

ECOC Conference

Geneva, September 18th, 2011

1

INTRODUCTION

First Year of LHC from the network perspective

2

WLCG Worldwide Resources

3

Today >140 sites>250k CPU cores>150 PB disk

Today we have 49 MoU signatories, representing 34 countries:Australia, Austria, Belgium, Brazil, Canada, China, Czech Rep, Denmark, Estonia, Finland, France, Germany, Hungary, Italy, India, Israel, Japan, Rep. Korea, Netherlands, Norway, Pakistan, Poland, Portugal, Romania, Russia, Slovenia, Spain, Sweden, Switzerland, Taipei, Turkey, UK, Ukraine, USA.

WLCG Collaboration StatusTier 0; 11 Tier 1s; 68 Tier 2 federations

In addition to WLCG, O(300) Tier-3 sites, not shown

Data and Computing Models

4

The Evolving MONARC Picture: Circa 2003

From Ian Bird, ICHEP 2010

Variations by experiment

The models are based

on the MONARC

model

Now 10+ years old

Circa 1996

The LHC Optical Private Network Serving Tier0 and Tier1 sites

• Dedicated network resources for Tier0 and Tier1 data movement• Layer 2 overlay on R&E

infrastructure• 130 Gbps total Tier0-Tier1

capacity• Simple architecture

– Point-to-point Layer 2 circuits– Flexible and scalable topology

• Grew organically– From star to partial mesh

•Open to technology choices• have to satisfy requirements• OC-192/SDH-64, EoMPLS,

OTN-3• Federated governance model

– Coordination between stakeholders

5

6

MB/sper day

6 GB/s

Peaks of 10 GB/s reached

~2 GB/s(design)

Grid-based analysis in Summer 2010: >1000 different users; >15M analysis jobs

The excellent Grid performance has been crucial for fast release of physics results. E.g.: ICHEP: the full data sample taken until Monday was shown at the conference Friday

2010 Worldwide data distribution and analysis (F.Gianotti)Total throughput of ATLAS data through the Grid: 1st January November.

CMS Data Movements (2010) (All Sites and Tier1-Tier2)

7

1 hour average: to 3.5 GBytes/s

Th

rou

gh

pu

t

[GB

y/s]

3

4

2

Th

rou

gh

pu

t

[GB

y/s]

1

2

0

1.5

2.5

0.5

Daily average total rates reach over

2 GBytes/s

1

0

1

2

0

1.5

0.5

120 Days June-October 2010

Daily average T1-T2 rates reach

1-1.8 GBytes/s

132 Hours in Oct. 2010

6/19 7/03 7/17 7/31 8/14 8/28 9/11 9/25 10/9 6/23 7/07 7/21 8/4 8/18 9/1 9/15 9/29 10/13

10/6 10/7 10/8 10/9 10/10

Tier2-Tier2 ~25% of Tier1-Tier2

Traffic

To ~50% during Dataset

Reprocessing & Repopulation

THE RESEARCH AND EDUCATION NETWORKING LANDSCAPE

Selected representative examples

8

GEANT Pan-European Backbone

9

Dark Fiber Core Among 19 Countries:

Austria Belgium Croatia Czech Republic Denmark Finland France Germany Hungary Ireland Italy Netherlands Norway Slovakia Slovenia Spain Sweden Switzerland United Kingdom

34 NRENs, ~40M Users; 50k km Leased Lines 12k km Dark Fiber; Point to Point ServicesGN3 Next Gen. Network Started in June 2009

SURFNet & NetherLight: 8000 Km Dark FiberFlexible Photonic Infrastructure

5 Photonic Subnets

λ Switching 10G, 40G; 100G Trials

Fixed or Dynamic

Lightpaths for

LCG, GN3, EXPRES

DEISALOFAR CineGrid

5 Photonic Subnets

λ Switching 10G, 40G; 100G Trials

Fixed or Dynamic

Lightpaths for

LCG, GN3, EXPRES

DEISALOFAR CineGrid Cross Border Fiber: to Belgium, on to CERN

(1650km); to Germany: X-Win, On to NORDUnet;Cross Border Fiber: to Belgium, on to CERN

(1650km); to Germany: X-Win, On to NORDUnet; Erik-Jan Bos

GARR-X in Italy: Dark Fiber Network Supporting LHC Tier1 and Nat’l Tier2 Centers

GARR-X

10G Links Among Bologna Tier1

& 5 Tier2s

Adding 5 More Sites at 10G

2 x 10G Circuits to the LHCOPNOver GEANT

and to Karlsruhe Via Int’l Tier2 – Tier1 Circuits

GARR-X

10G Links Among Bologna Tier1

& 5 Tier2s

Adding 5 More Sites at 10G

2 x 10G Circuits to the LHCOPNOver GEANT

and to Karlsruhe Via Int’l Tier2 – Tier1 Circuits

M. MarlettaCross Border Fibers to Karlsruhe (Via CH, DE)Cross Border Fibers to Karlsruhe (Via CH, DE)

US: DOE ESnet

12

Current ESnet4 Topology: Multi-10G backbone

SDN nodeIP router node10G linkMajor site v

DOE Esnet – 100Gbps Backbone Upgrade

13

100G nodeRouter node100G linkMajor site v

ESnet5 100G Backbone, Q4 2012First deployment started Q3 2011

US LHCNetNon-stop Operation; Circuit-oriented Services

14

Performance enhancingStandard

Extensions:VCAT, LCAS

USLHCNet, ESnet, BNL

& FNAL:Facility,

equipment and link

redundancy

Core: Optical multiservice

Switches

Dynamic circuit-oriented network services with BW guarantees, with robust fallback at layer 1: Hybrid optical network

Dark Fiber in NREN Backbones 2005 – 2010Greater or Complete Reliance on Dark Fiber

TERENA Compendium 2010: www.terena.org/activities/compendium/

2005 2010

Cross Border Dark Fiber in EuropeCurrent and Planned: Increasing Use

TERENA Compendium

2010

Global Lambda Integrated Facility

A Global Partnership of R&E Networks and Advanced Network R&D Projects Supporting HEP

http://glif.is

GLIF 2010 Map – Global View

GLIF 2010 Map North America

~16 10G Trans- Atlantic Links

in 2010

GLIF Open Lightpath Exchanges:MoscowLight, CzechLight, CERNLight, NorthernLight

NetherLight, UKLight

GLIF Open Lightpath Exchanges:MoscowLight, CzechLight, CERNLight, NorthernLight

NetherLight, UKLight

GLIF 2010 Map: European ViewR&E Networks, Links and GOLEs

Open Exchange Points: NetherLight Example

3 x 40G, 30+ 10G Lambdas, Use of Dark Fiber

Convergence of Many Partners on Common Lightpath Concepts Internet2, ESnet, GEANT, USLHCNet; nl, cz, ru, be, pl, es, tw, kr, hk, in, nordicConvergence of Many Partners on Common Lightpath Concepts

Internet2, ESnet, GEANT, USLHCNet; nl, cz, ru, be, pl, es, tw, kr, hk, in, nordic

LHC NETWORKING - BEYOND LHCOPN

21

Computing Models Evolution

• Moving away from the strict MONARC model• Introduced gradually since 2010• 3 recurring themes:

– Flat(ter) hierarchy: Any site can use any other site as source of data

– Dynamic data caching: Analysis sites will pull datasets from other sites “on demand”, including from Tier2s in other regions

• Possibly in combination with strategic pre-placement of data sets– Remote data access: jobs executing locally,

using data cached at a remote site in quasi-real time

• Possibly in combination with local caching

• Variations by experiment

22

LHC Open Network Environment

• So far, T1-T2, T2-T2, and T3 data movements have been using General Purpose Network infrastructure– Shared resources (with other science fields)– Mostly best effort service

• Increased reliance on network performance need more than best effort• Separate large LHC data flows from routed R&E GPN

• Collaboration on global scale, diverse environment, many parties– Solution to be Open, Neutral and Diverse – Agility and Expandability

• Scalable in bandwidth, extent and scope• Organic activity, growing over time according to needs• Architecture:

– Switched Core, Routed Edge– Core: Interconnecting trunks between Open Exchanges– Edge: Site Border Routers, or BRs of regional aggregation networks

• Services: Multipoint, static point-to-point, dynamic point-to-point

23

LHCONE High-Level Architecture Overview

24

LHCONE Conceptual

Diagram

LOOKING FORWARD: NEW NETWORK SERVICES

25

Characterization of User Space

26

Cees de Laat; http://ext.delaat.net/talks/cdl-2005-02-13.pdf

This is where LHC users are

27

David Foster, 1st TERENA ASPIRE Workshop, May 2011

The Case for Dynamic Circuits in LHC Data Processing

• Data models do not require full-mesh @ full-rate connectivity @ all times• On-demand data movement will augment and partially replace static pre-

placement Network utilisation will be more dynamic and less predictable• Performance expectations will not decrease

– More dependence on the network, for the whole data processing system to work well!

• Need to move large data sets fast between computing sites– On-demand: caching– Scheduled: pre-placement– Transfer latency is important

• Network traffic far in excess of what was anticipated• As data volumes grow rapidly, and experiments rely increasingly on the

network performance - what will be needed in the future is– More bandwidth – More efficient use of network resources– Systems approach including end-site resources and software stacks

• Note: Solutions for the LHC community need global reach28

Dynamic Bandwidth Allocation

• Will be one of the services to be provided in LHCONE• Allows to allocate network capacity on as-needed basis

– Instantaneous (“Bandwidth on Demand”), or – Scheduled allocation

• Significant effort in R&E Networking community– Standardisation through OGF (OGF-NSI, OGF-NML)

• Dynamic Circuit Service is present in several advanced R&E networks – SURFnet (DRAC)– ESnet (OSCARS)– Internet2 (ION)– US LHCNet (OSCARS)

• Planned (or in experimental deployment)– E.g. GEANT (AutoBahn), RNP (OSCARS/DCN), …

• DYNES: NSF funded project to extend hybrid & dynamic network capabilities to campus & regional networks – In first deployment phase; fully operational in 2012

29

US Example: DYNES Project

• NSF-funded project: DYnamic NEtwork System• What is it?

– A nationwide cyber-instrument spanning up to ~40 US universities and ~14 Internet2 connectors

– Extends Internet2s ION service into regional networks and campuses, based on ESnet’s OSCARS implementation of IDC protocol

• Who is it?– A collaborative team including Internet2, Caltech, University of Michigan, and

Vanderbilt University – Community of regional networks and campuses– LHC, astrophysics community, OSG, WLCG, other virtual organizations

• The goals– Support large, long-distance scientific data flows in the LHC, other leading

programs in data intensive science (such as LIGO, Virtual Observatory, and other large scale sky surveys), and the broader scientific community

– Build a distributed virtual instrument at sites of interest to the LHC but available to R&E community generally

30http://www.internet2.edu/dynes

DYNES System Description

• AIM: extend hybrid & dynamic capabilities to campus & regional networks – A DYNES instrument must provide two basic capabilities at the Tier 2s, Tier3s and

regional networks:

1. Network resource allocation such as bandwidth to ensure transfer performance

2. Monitoring of the network and data transfer performance

• All networks in the path require the ability to allocate network resources and monitor the transfer. This capability currently exists on backbone networks such as Internet2 and ESnet, but is not widespread at the campus and regional level– In addition Tier 2 & 3 sites require: 3. Hardware at the end sites capable of

making optimal use of the available network resources

31

Two typical transfers that DYNES supports: one Tier2 - Tier3 and

another Tier1-Tier2.

The clouds represent the network domains involved in such a transfer.

Summary

• LHC Computing models rely on efficient high-throughput data movement between computing sites (Tier0/1/2/3)

• Close collaboration between the LHC and R&E networking communities– Regional, National, International

• LHCOPN (LHC Optical Private Network):– Layer 2 overlay network, dedicated resources for Tier0 and Tier1 centres– Very successful operation

• LHCONE (LHC Open Network Environment):– New initiative to provide reliable services to ALL LHC computing sites (Tier 0-3)– Being developed as collaboration between LHC community and the Research

and Education Networks world-wide– User driven, organic growth– Current architecture is built on switched core with routed edge– Will provide advanced network services with dynamic bandwidth allocation

32

THANK YOU!

Artur.Barczyk@cern.ch

33