CHEP ‘04 CHEP ‘04 LHC Computing Models and Data Challenges LHC Computing Models and Data Challenges LHC Computing Models and Data LHC Computing Models and Data Challenges Challenges David Stickland David Stickland Princeton University Princeton University For the LHC Experiments: For the LHC Experiments: ALICE, ATLAS, CMS and LHCb ALICE, ATLAS, CMS and LHCb (Though with entirely my own bias (Though with entirely my own bias and errors) and errors)
Transcript
CH
EP
‘04
CH
EP
‘04
LH
C C
om
pu
tin
g M
od
els
an
d D
ata
Ch
allen
ges
LH
C C
om
pu
tin
g M
od
els
an
d D
ata
Ch
allen
ges
LHC Computing Models and Data LHC Computing Models and Data ChallengesChallenges
David SticklandDavid Stickland
Princeton UniversityPrinceton University
For the LHC Experiments:For the LHC Experiments:
ALICE, ATLAS, CMS and LHCbALICE, ATLAS, CMS and LHCb
(Though with entirely my own bias (Though with entirely my own bias and errors)and errors)
David SticklandPrinceton Univ.30 Sept 2004
Page 2/32
CH
EP
‘04
CH
EP
‘04
LH
C C
om
pu
tin
g M
od
els
an
d D
ata
Ch
allen
ges
LH
C C
om
pu
tin
g M
od
els
an
d D
ata
Ch
allen
ges
Outline
Elements of a Computing Model
Recap of LHC Computing Scales
Recent/Current LHC Data Challenges
Critical Issues
David SticklandPrinceton Univ.30 Sept 2004
Page 3/32
CH
EP
‘04
CH
EP
‘04
LH
C C
om
pu
tin
g M
od
els
an
d D
ata
Ch
allen
ges
LH
C C
om
pu
tin
g M
od
els
an
d D
ata
Ch
allen
ges
The Real Challenge…
Quoting from David Williams’ Opening Talk– Computing hardware isn’t the biggest challenge
– Enabling the collaboration to bring their combined intellect to bear on the problems is the real challenge
Empowering the intellectual capabilities in very large collaborations to contribute to the analysis of the new energy frontier
– Not (just) from sociological/political perspective
– You never know where the critical ideas or work will come from
– The simplistic vision of central control is elitist and illusory
David SticklandPrinceton Univ.30 Sept 2004
Page 4/32
CH
EP
‘04
CH
EP
‘04
LH
C C
om
pu
tin
g M
od
els
an
d D
ata
Ch
allen
ges
LH
C C
om
pu
tin
g M
od
els
an
d D
ata
Ch
allen
ges
Goals of An (Offline) Computing Model
Worldwide Collaborator access and ability to contribute to the analysis of the experiment:
Safe Data Storage
Feedback to the running experiment
Reconstruction according to priority scheme with graceful fall back solutions
– Introduce no Deadtime
Optimum (Or at least acceptable) usage of Computing Resources
Efficient Data Management
David SticklandPrinceton Univ.30 Sept 2004
Page 5/32
CH
EP
‘04
CH
EP
‘04
LH
C C
om
pu
tin
g M
od
els
an
d D
ata
Ch
allen
ges
LH
C C
om
pu
tin
g M
od
els
an
d D
ata
Ch
allen
ges
Where are LHC Computing Models today?
In a state of flux!– Last chance to influence initial purchasing planning in next 12 months
Basic principles – were contained in MONARC and went into the first assessment of LHC
computing requirements– Aka “Hoffmann Report”; Instrumental in establishment of LCG
But, now we have two critical milestones to meet– Computing Model Papers (Dec 2004)– Experiment and LCG TDR’s (July 2005)
And, we more or less know what will, or won’t, be ready in time
– Restrict enthusiasm to attainable goals
Process now to review models in all four experiments.– Very interested in running experiment experiences, (This Conference!)– Need the maximum expertise and experience included in these final
CM’s
David SticklandPrinceton Univ.30 Sept 2004
Page 6/32
CH
EP
‘04
CH
EP
‘04
LH
C C
om
pu
tin
g M
od
els
an
d D
ata
Ch
allen
ges
LH
C C
om
pu
tin
g M
od
els
an
d D
ata
Ch
allen
ges
LHC Computing Scale (Circa 2008)
CERN T0/T1– Disk Space [PB] 5– Mass Storage Space [ PB] 20– Processing Power [MSI2K] 20– WAN [10Gb/s] ~5?
Tier-1s (Sum of ~10)– Disk Space [PB] 20– Mass Storage Space [ PB] 20– Processing Power [MSI2K] 45– WAN [10Gb/s/Tier-1]
~1?
Tier-2s (Sum of ~40)– Disk Space [PB] 12– Mass Storage Space [ PB] 5– Processing Power [MSI2K] 40– WAN [10Gb/s/Tier-2]
Move a copy of the raw data away from CERN in “real-time”
– Second secure copy• 1 copy at CERN • 1 copy spread over N sites
– Flexibility. • Serve raw data even if Tier-0
saturated with DAQ
– Ability to run even primary reconstruction offsite
Streaming online and offline– (Maybe not a common theme
yet)
Tier-1 centers in-line to Online and Tier-0
Simulation at T2 centers– Except LHCb, if simulation load
remains high, use Tier-1
ESD Distributed n copies over N Tier-1 sites
– Tier-2 centers run complex selections at Tier-1, download skims
AOD Distributed to all (?) tier-2 centers
– Maybe not a common theme. • How useful is AOD, how early in
LHC?• Some Run II experience
indicating long term usage of “raw” data
Horizontal Streaming– RAW, ESD, AOD,TAG
Vertical Streaming– Trigger streams, Physics
Streams, Analysis Skims
David SticklandPrinceton Univ.30 Sept 2004
Page 9/32
CH
EP
‘04
CH
EP
‘04
LH
C C
om
pu
tin
g M
od
els
an
d D
ata
Ch
allen
ges
LH
C C
om
pu
tin
g M
od
els
an
d D
ata
Ch
allen
ges
Purpose and structure of ALICE PDC04
Test and validate the ALICE Offline computing model:– Produce and analyse ~10% of the data sample collected in a
standard data-taking year– Use the entire ALICE off-line framework: AliEn, AliRoot, LCG, PROOF…– Experiment with Grid enabled distributed computing– Triple purpose: test of the middleware, the software and physics
analysis of the produced data for the Alice PPR
Three phases– Phase I - Distributed production of underlying Pb+Pb events with
different centralities (impact parameters) and of p+p events– Phase II - Distributed production mixing different signal events into
the underlying Pb+Pb events (reused several times)– Phase III – Distributed analysis
Principles:– True GRID data production and analysis: all jobs are run on the GRID,
using only AliEn for access and control of native computing resources and, through an interface, the LCG resources
The ALICE DC04 started out with (almost unrealistically) ambitious objectives
They are coming very close to reach these objectives and LCG has played an important role.
They are ready and willing to move to gLite as soon as possible and contribute to its evolution with their feedback
David SticklandPrinceton Univ.30 Sept 2004
Page 17/32
CH
EP
‘04
CH
EP
‘04
LH
C C
om
pu
tin
g M
od
els
an
d D
ata
Ch
allen
ges
LH
C C
om
pu
tin
g M
od
els
an
d D
ata
Ch
allen
ges
Consider DC2 as a three-part operation:– part I: production of simulated data (July-September 2004)
• running on “Grid”• Worldwide
– part II: test of Tier-0 operation (November 2004)• Do in 10 days what “should” be done in 1 day when real data-
taking start• Input is “Raw Data” like• output (ESD+AOD) will be distributed to Tier-1s in real time for
analysis
– part III: test of distributed analysis on the Grid • access to event and non-event data from anywhere in the world
both in organized and chaotic ways
Requests– ~30 Physics channels ( 10 Millions of events)
– Several millions of events for calibration (single particles and physics samples)
ATLAS-DC2 operation
David SticklandPrinceton Univ.30 Sept 2004
Page 18/32
CH
EP
‘04
CH
EP
‘04
LH
C C
om
pu
tin
g M
od
els
an
d D
ata
Ch
allen
ges
LH
C C
om
pu
tin
g M
od
els
an
d D
ata
Ch
allen
ges
LCG NG Grid3 LSF
LCGexe
LCGexe
NGexe
G3exe
LSFexe
super super super super super
prodDBdms
RLS RLS RLS
jabber jabber soap soap jabber
Don Quijote
Windmill
Lexor
AMI
CaponeDulcinea
ATLAS Production system
David SticklandPrinceton Univ.30 Sept 2004
Page 19/32
CH
EP
‘04
CH
EP
‘04
LH
C C
om
pu
tin
g M
od
els
an
d D
ata
Ch
allen
ges
LH
C C
om
pu
tin
g M
od
els
an
d D
ata
Ch
allen
ges
CPU usage & Jobs
NorduGrid
50%
12%
8%
3%
3%
3%
3%
2%
2%
2%
2%
2%
2%
2%
2%
0%
0%
SWEGRID
brenta.ijs.si
benedict.aau.dk
hypatia.uio.no
farm.hep.lu.se
fire.ii.uib.no
fe10.dcsc.sdu.dk
lxsrv9.lrz-muenchen.de
atlas.hpc.unimelb.edu.au
grid.uio.no
lheppc10.unibe.ch
morpheus.dcgc.dk
genghis.hpc.unimelb.edu.au
charm.hpc.unimelb.edu.au
lscf.nbi.dk
atlas.fzk.de
grid.fi.uib.no
ATLAS DC2 - CPU usage
41%
30%
29%
LCG
NorduGrid
Grid3
ATLAS DC2 - Grid3 - September 7
7%
10%
13%
4%
5%
0%
10%5%
1%
1%
5%
4%
13%
1%
4%
0%
18%
1%
BNL_ATLAS
BNL_ATLAS_BAK
BU_ATLAS_Tier2
CalTech_PG
FNAL_CMS
FNAL_CMS2
IU_ATLAS_Tier2
PDSF
Rice_Grid3
SMU_Physics_Cluster
UBuffalo_CCR
UCSanDiego_PG
UC_ATLAS_Tier2
UFlorida_PG
UM_ATLAS
UNM_HPC
UTA_dpcc
UWMadison
ATLAS DC2 - LCG - September 7
0%
2%
14%
3%
1%
3%
9%
8%
3%2%5%1%4%
1%
1%
3%
0%
1%
4%1%
0%
12%
0%
1%
1%
2%
10%
1% 4%
2%1%
1%
1%
at.uibk
ca.triumf
ca.ualberta
ca.umontreal
ca.utoronto
ch.cern
cz.golias
cz.skurut
de.fzk
es.ifae
es.ific
es.uam
fr.in2p3
it.infn.cnaf
it.infn.lnl
it.infn.mi
it.infn.na
it.infn.na
it.infn.roma
it.infn.to
it.infn.lnf
jp.icepp
nl.nikhef
pl.zeus
ru.msu
tw.sinica
uk.bham
uk.ic
uk.lancs
uk.man
uk.rl
uk.shef
uk.ucl
David SticklandPrinceton Univ.30 Sept 2004
Page 20/32
CH
EP
‘04
CH
EP
‘04
LH
C C
om
pu
tin
g M
od
els
an
d D
ata
Ch
allen
ges
LH
C C
om
pu
tin
g M
od
els
an
d D
ata
Ch
allen
ges
ATLAS DC2 Status
Major efforts in the past few months
– Redesign of the ATLAS Event Data Model and Detector Description
– Integration of the LCG components (G4; POOL; …)
– Introduction of the Production System
• Interfaced with 3 Grid flavors (and “legacy” systems)
Delays in all activities have affected the schedule of DC2
– Note that Combined Test Beam is ATLAS 1st priority
– And DC2 schedule was revisited
• To wait for the readiness of the software and of the Production system
DC2– About 80% of the Geant4
simulation foreseen for Phase I has been completed using only Grid and using the 3 flavors coherently;
– The 3 Grids have been proven to be usable for a real production and this is a major achievement
BUT– Phase I progressing slower than
expected and it’s clear that all the involved elements (Grid middleware; Production System; deployment and monitoring tools over the sites) need improvements
– It is a key goal of the Data Challenges to identify these problems as early as possible.
David SticklandPrinceton Univ.30 Sept 2004
Page 21/32
CH
EP
‘04
CH
EP
‘04
LH
C C
om
pu
tin
g M
od
els
an
d D
ata
Ch
allen
ges
LH
C C
om
pu
tin
g M
od
els
an
d D
ata
Ch
allen
ges
Testing CMS Computing Model in DC04 Focused on organized (CMS-managed) data flow/access
Functional DST with streams for Physics and Calibration– DST size ok, almost usable by “all” analyses; (new version ready now)
Tier-0 farm reconstruction– 500 CPU. Ran at 25Hz. Reconstruction time within estimates.
Tier-0 Buffer Management and Distribution to Tier-1’s– TMDB: a CMS-built Agent system communicating via a Central Database.– Manages dynamic dataset “state”, not a file catalog
Tier-1 Managed Import of Selected Data from Tier-0– TMDB system worked.
Tier-2 Managed Import of Selected Data from Tier-1– Meta-data based selection ok. Local Tier-1 TMDB ok.
Real-Time analysis access at Tier-1 and Tier-2– Achieved 20 minute latency from Tier 0 reconstruction to job launch at Tier-
1 and Tier-2
Catalog Services, Replica Management– Significant performance problems found and being addressed
David SticklandPrinceton Univ.30 Sept 2004
Page 22/32
CH
EP
‘04
CH
EP
‘04
LH
C C
om
pu
tin
g M
od
els
an
d D
ata
Ch
allen
ges
LH
C C
om
pu
tin
g M
od
els
an
d D
ata
Ch
allen
ges
PICBarcelona
FZKKarlsruhe
CNAFBologna
RALOxford
IN2P3Lyon
T1
T1
T1
T1
T1 T0
FNALChicago
T1
DC04 Data Challenge Focused on organized (CMS-managed) data flow/access
T0 at CERN in DC04– 25 Hz Reconstruction– Events filtered into streams– Record raw data and DST– Distribute raw data and DST to
T1’s
T1 centres in DC04– Pull data from T0 to T1 and
store– Make data available to PRS– Demonstrate quasi-realtime
analysis of DST’s
T2 centres in DC04– Pre-challenge production at >
30 sites– Modest tests of DST analysis
T2
Legnaro
T2 T2
CIEMATMadrid
Florida
T2
ICLondon
T2
Caltech
David SticklandPrinceton Univ.30 Sept 2004
Page 23/32
CH
EP
‘04
CH
EP
‘04
LH
C C
om
pu
tin
g M
od
els
an
d D
ata
Ch
allen
ges
LH
C C
om
pu
tin
g M
od
els
an
d D
ata
Ch
allen
ges
Tier-2Tier-2
Physicist
T2T2storagestorage
ORCALocal Job
Tier-2Tier-2
Physicist
T2T2storagestorage
ORCALocal Job
Tier-1Tier-1Tier-1agent
T1T1storagestorage
ORCAAnalysis
Job
MSS
ORCAGrid Job
Tier-1Tier-1Tier-1agent
T1T1storagestorage
ORCAAnalysis
Job
MSS
ORCAGrid Job
DC04 layout
Tier-0Tier-0
Castor
IBIB
fake on-lineprocess
RefDB
POOL RLScatalogue
TMDB
ORCARECO
Job
GDBGDBTier-0
data distributionagents
EBEB
LCG-2Services
Tier-2Tier-2
Physicist
T2T2storagestorage
ORCALocal JobTier-1Tier-1
Tier-1agent
T1T1storagestorage
ORCAAnalysis
Job
MSS
ORCAGrid Job
David SticklandPrinceton Univ.30 Sept 2004
Page 24/32
CH
EP
‘04
CH
EP
‘04
LH
C C
om
pu
tin
g M
od
els
an
d D
ata
Ch
allen
ges
LH
C C
om
pu
tin
g M
od
els
an
d D
ata
Ch
allen
ges
Next Steps
Physics TDR requires physicist access to DC04 data
