INSTR021 Computing in High Energy Physics INSTR02 March 6 2002 Ren é Brun CERN.

INSTR02 1

Computing in High Energy Physics

INSTR02March 6 2002

René Brun

CERN

Rene Brun Computing in HEP / INSTR02 2

Computing in High Energy Physics

A wide subject I want to focus on what I know best

Offline Computing and its relations with Online

Data Storage & Management Software Frameworks Distributed Data Access & Analysis the GRID projects

Show Evolution and discuss trends


The OLD Model

Data StructuresBanks Managers

ZEBRA BOS

Experimentspecific software

in FortranEvent Generators

Pythia, Isajet

CERNLIB

Geant3 PAW


Understanding Trends

Looking backward (past 20 years) is necessary to understand the current trends.

I am not reading in the crystal ball. My perception is subjective and biaised by my current involvement in the ROOT project. View is LHC-centric.

I will not risk myself beyond the LHC startup (or more precisely 2006/2007).

Some examples from the ALICE collaboration


Move to OO Programming The first attempts to introduce the OO concepts in HEP software were

made in 1989. These attempts were unsuccessful for many reasons:

OO languages were not mature and not widely available.

The performance of compilers could not compete with the Fortran compilers.

The introduction of a new system requires that this new system provides all the facilities of the previous system in a stable and proven environment. This is a chicken and the egg problem. Convincing the silent majority to adopt a new tool or framework is a challenging work.

The problem of Object Persistency had been totally underestimated or a wrong solution assumed.

Last, but not least, the introduction of a new system could only be done with the strong support of a major laboratory and a long term commitment. The situation at CERN had been very confusing.


LanguagesIn 1991, my boss said:

“I do not know what will be the future language

but I know that it will be called Fortran”

Fortran: rapidly vanishing

C++: the main stream language

Java: much less used than expected

C# the Microsoft big ?


Java: The end ?

Java is the current maintream language for Web-based

applications. Still not really portable. Java applications seen as very slow. MS wants to kill Java.

JAS: Interesting features, but not used because expmts went to C++ and Linux.


ROOT Downloads

123,000 binariesdownload

1,300,000 clicksper month

35,000 docsin 12 months

2200 reg usersin roottalk


Productivity

C++ is a very powerful language, but extremely difficult to learn.

Transition from the Fortran world has been very painful.

Slow process because basic libraries not available in the early days


A bit of History

1994/95 Official line projects start RD44 (Geant4), RD45 (Objectivity)

Jan 1995. ROOT project starts in NA49,ALICE Sep 1995. LHC++ (CERNLIB replacement) starts Oct 1995. JAS (Java Analysis Studio) starts (SLAC) Nov 1998. CDF adopts ROOT, followed by STAR, RHIC,etc Jan 2000. LHC++ renamed to Anaphe Oct 2001. CMS changes its baseline from Objectivity to

ROOT Feb 2002. ROOT is used by most HEP experiments LCG/SC2 committees. ROOT official support

In 1994, fundamental divergence of opinions in Application Software group in IT. The PAW/Geant3 team is dismantled.

Fortra

n9

0??

?

C++,Commercial Software

C++,

Open

Sourc

e

ROOT


Current Situation

Rapidly evolving from the Old Model to the New Model described later.

Painful transition to C++ in most major collaborations.

A few collaborations have experience with ODBMS systems (ie Objectivity)

But the days of Objectivity in HEP are counted.

Some limited experience with Geant4 Move to ROOT


Situation in the US

BaBar: pioneers with Objy. Complex situation.Kanga/Root used in small labs. Analysis with PAW. Growing Root community. G3-->G4

Glast: Use Gaudi + Root (G3 ->G4?) STAR: Root + MySQL + G3 Phenix: Root + Objy(catalog) +G3 Phobos: Root + Oracle + G3 JLAB: Complex situations with the small

expmts. Root in the main exp. G3. Attempts to use G4.


Situation in the US CDF: Root-based I/O and analysis (online +

offline). Final transition from Fortran to C++. File catalog in Oracle. G3, attempts to use G4

D0: home-grown I/O system (using CINT). Move from PAW to Root for data analysis. SAM (Oracle) catalog. G3

Minos: Root-based framework. G3 BTeV: ? Attempts to use G4 Miniboon: Fortran ?


Situation in Japan

Belle: Home grown tools in C++: - non-OO Data Management System Panther ADAMO-like structure with a C++ interface - Analysis Framework : "B.A.S.F." module and path structure dynamic link of modules and I/O drivers event-by-event parallel processing capability on

SMP - Communication tool over network : "NSM" shared memory/message passing capability over

network to be used for slow control

Linear Collider: C++/Root-based. G3


Situation at DESY

H1: Moving from Fortran-BOS to C++/Root in a BOS-compatible way. Use Oracle, G3.

ZEUS: Moving from Fortran to C++. Have used Objectivity. PAW analysis. Root growing. G3

Hera-B: Has moved from Fortran to C++/Root

Tesla: Moving from Fortran G3 to C++/G4


Situation with CERN non-LHC

Aleph, Opal, Delphi, L3: Fortran/Zebra/PAW/G3.

Compass: C++ home-grown system + Objectivity. PAW-->Root for analysis. G3

Harp: Objectivity, G4 AMS: C++ home-grown system. First

guinea pigs with Objectivity. Use PAW, Oracle and Root, G3

NA60: Root (Alice clone)


Situation with LHC

ALICE: Root framework + MySQL. G3 and G4. Alien. Abandoning G4. Interface to Fluka

ATLAS: Develop Athena framework (based on Gaudi). Objectivity & Root. G3. Experience with G4.

CMS: Orca/Cobra frameworks. Was based on Objectivity. Switching to Root. G3. Experience with G4.

LHCB: Gaudi framework with Root I/O. G3, starting with G4.


1 billion people surfing the

Web

How Much Data is Involved?

105

104

103

102

Level 1 Rate (Hz)

High Level-1 Trigger(1 MHz)

High No. ChannelsHigh Bandwidth(500 Gbit/s)

High Data Archive(5 PetaBytes/year)10 Gbits/s in Data base

LHCB

KLOE

HERA-B

CDF II

CDF

H1ZEUS

UA1

LEP

NA49ALICE

Event Size (bytes)

104 105 106

ATLASCMS

106

107

STAR


ALICE Event/100

Front View of a simulatedevent with only 1/100 of the

expected multiplicity


ALICE Event/100Side View of a simulatedevent with only 1/100 of the

expected multiplicity

Estimated size of one raw event = 40 Mbytes

Simulated event with hits = 1.5 Gbytes

Time to simulate one event = 24 hours

After L3, the DAQ will generate 1.25 GigaBytes/second

2 PetaBytes/year


LHC Computing - a Multi-Tier Model

Department

Desktop

CERN – Tier 0(CERN - Tier 1)

Tier 1 X Y

Z622 M

bps2.5 Gbps

622 M

bp

s

155

mbp

s155 mbps

Tier2 Lab a

Uni b Lab c

Uni n Organising Software:

"Grid-Middleware"

"Transparent" user access to applications and all data

'X''Y''Z': RAL, IN2P3, FNAL, BNL, FZK(?), . . .


People

Number of peopleand world-wide collaborations

have a big impacton the Computing

strategy


Worldwide Software Development

Code Management tools Is CVS sufficient ?

Concurrent development implications side-effects of unstable source code concepts of old/pro/new obsolete

Nightly builds Code inspection/Quality checks procedures

Bug reporting system HyperNews Web-based documentation & information


Move to OO programming

Data structuresin Fortran

common blocks

Bankssystems

Zebra,BosBanks-like

systems in C/C++Hand-written

I/O converters

True ObjectsCollections in C++

with built-inObject persistency and

automatic schema evolution

Typical scenario


Dynamic Linking

ApplicationExecutable Module

Experimentlibraries

Userlibraries

Generallibraries

A Shared Library can be linked dynamically to a running executable module

A Shared Library facilitates the development and maintenance phases


Whiteboard Data Communication

Class 1 Class 2

Class 3

Class 4

Class 5Class 6

Class 7

Class 8


Generic task handling

Det

Det

Det

Det

Det

Det

DetTaskTask

Task

TaskTask

Task

TaskTask

Task

TaskTask

Task

TaskTask

Task

TaskTask

Task

TaskTask

Task


event

From a simple nested parenthesis structure

((..)(..)(((..)(..)(…..))))To complex Objects graphs with internal and external references

In KBytes

BA


Lines of code

G4: 600,000 C++ lines

ROOT: 700,000 C++ lines

700 classes

30 sub-systems

ALICE: 600,000 C++ lines

+ROOT + G4 +..+..


A CORE system

Objects Dictionary: Introspection

GUI: Interpreter I/O subsystem

Graphics, Histograms Virtual file Catalog

High Level apps GRID toolkits

Experiment Software

Event Store Data bases


Data Bases: model-1

Put everything in an Object Data base like Objectivity or Oracle 9i

Choice of RD45 project Many experiments initially following this

line Abandonned by most experiments

recently Solution not suited for interactive analysis


Data Bases: model-2

Put write-once data in an object store like ROOT in Streamer mode

Use a RDBMS for : Run/Event catalogs Geometry, calibrations eg with ROOT<->Oracle interface

http://www.phenix.bnl.gov/WWW/publish/onuchin/rooObjy/

or with ROOT <-> Objectivity interface http://www.phenix.bnl.gov/WWW/publish/onuchin/RDBC/

Use ROOT split/no-split mode for data analysis

Combining2 technologies

ROOT

Oracle


The LHC Computing Grid Project

Next week at CERN


A ROOT-based CORE system

Objects Dictionary: Introspection

GUI: Interpreter I/O subsystem

Graphics, Histograms Virtual file Catalog

High Level apps GRID toolkits

Experiment Software

Event Store Data bases

agreed


ROOT + RDBMS Model

histograms

Calibrations

Geometries

Run/FileCatalog

Trees

Event Store

ROOTfiles

OracleMySQL


Memory <--> TreeThe Tree entry serial number

0123456789101112131415161718

T.Fill()

T.GetEntry(6)

T

Memory


Tree Friends

0123456789101112131415161718

0123456789101112131415161718

0123456789101112131415161718

Public

read

Public

read

User

Write

Entry # 8


Chains of Trees

0 0 0 0 0 00

11

0

4 52

45 4

7

0 12 17 23 26 32 37 45 49

TChain ch(“T”);

ch.Add(“f0.root”);








ch.GetEntry(28);

Binary search in table above

find slot 4, local entry 2

T.GetEntry(2) in f4.root

ch.GetEntryWithIndex(12,567);

0 7654321

f0 f2 f3 f4 f5 f6 f7f1


Automatic Schema Evolution


Auto Schema Evolution (2)


Self-describing files

Dictionary for persistent classes written to the file.

ROOT files can be read by foreign readers (JAS)

Support for Backward and Forward compatibility

Files created in 2002 must be readable in 2015

Classes (data objects) for all objects in a file can be regenerated via TFile::MakeProjectRoot >TFile f(“demo.root”);

Root > f.MakeProject(“dir”,”*”,”new++”);


Online/Offline

Groups traditionally separated. Growing synergy dictated by common

needs for data bases and event storage, GUI, graphics, analysis systems, L3 software, input from detector simulation programs, tools.

Inter-process objects and messages passing requires a common object dictionary.

Growing synergy imposed by the Data Challenges.

A good example: ALICE Data Challenges


Regional TIER 1 TIER 2

ALICE Data challenges

DAQ

ROOT I/OCERN TIER 0 TIER 1

Raw Data

Simulated Data GEANT3

GEANT4FLUKA

AliRoot

CASTOR

GRID

Performance Monitoring

ROOT

FileCatalogue


Writing to local diskMigration to tape

ALICE Data Challenge III Need to run yearly DC of increasing complexity and size to reach 1.25GB/s ADC III gave excellent system stability during 3 months

DATE throughput: 550 MB/s (max) 350 MB/s (ALICE-like) DATE+ROOT+CASTOR throughput: 120 MB/s, <85> MB/s 2200 runs, 2* 107 events, 86 hours, 54 TB DATE run 500 TB in DAQ, 200 TB in DAQ+ROOT I/O, 110 TB in CASTOR 105 files > 1GB in CASTOR and in MetaData DB

HP SMP’s: cost-effective alternative to inexpensive disk servers Online monitoring tools developed

0

500

1000

1500

2000

2500

3000

1999 2000 2001 2002 2003 2004 2005 2006

DAQMass Storage

0

50

100

150

200

250

300

350

1999 2000 2001

DAQ

Mass Storage (sust)

Mass Storage (peak)

MB

/s

MB

/s


CPU

Fantastic evolution, but not the only issue

Intel-compatible processors only ?

IA-64 processors ?

CERN units

Event-level parallelism


Disk

Disk bandwidth still a problem

IDE: 10->20 MB/s. Latency: > 5ms

GBytes

In 2007, cost of one Petabyte of disk or tape should be the same

Keep on disk DST/AOD as much as possible


GRID activities

Several Projects Starting in the US (iVDGL) and Europe (EDG)

No time to describe these projects.

Concentrate on one application

http://alien.cern.ch


ALICE GRID resources

Yerevan

CERN

Saclay

Lyon

Dubna

Capetown, ZA

Birmingham

Cagliari

NIKHEF

GSI

Catania

BolognaTorino

Padova

IRB

Kolkata, India

OSU/OSCLBL/NERSC

Merida

Bari

http://www.to.infn.it/activities/experiments/alice-grid

1000 physicists

37 people GRID-aware

21 institutions


ALICE GRID File Catalogue as a global file system on a RDB TAG Catalogue, as extension Secure Authentication

Interface to Globus available Central Queue Manager ("pull" vs "push" model)

Interface to EDG Resource Broker available Monitoring infrastructure

The CORE GRID functionality Automatic software installation with AliKit http://alien.cern.ch


ALIENalie n

SO AP Se rve r

AD M IN

P R O C ESSES

Autho r isatio nSe rvic e

allie n(she ll ,W e b)

C lie nt

D B IP ro xy se rve r

File C atalo gue

File C atalo gue

D BD rive r

File transpo r tSe rvic e

U se r Applic atio n(C /C + + /Java/P e r l)

SO AP C lie nt

D B SyncSe rvic e

D ISK

File catalogue : global file system

on top of relational database

Secure authentication service independent of underlying database

Central task queue

API

Services (file transport, sync)

Perl5

SOAP

Architecture

ALICEUSERS

ALICESIM

Tier1

ALICELOCAL

|--./| |--cern.ch/| | |--user/| | | |--a/| | | | |--admin/| | | | || | | | |--aliprod/| | | || | | |--f/| | | | |--fca/| | | || | | |--p/| | | | |--psaiz/| | | | | |--as/| | | | | || | | | | |--dos/| | | | | || | | | | |--local/

|--simulation/| |--2001-01/| | |--V3.05/| | | |--Config.C| | | |--grun.C

| |--36/| | |--stderr| | |--stdin| | |--stdout| || |--37/| | |--stderr| | |--stdin| | |--stdout| || |--38/| | |--stderr| | |--stdin| | |--stdout

| | | || | | |--b/| | | | |--barbera/

Files, commands (job specification) as well as job

input and output, tags and even binary package tar files are

stored in the catalogue

File catalogue

AD M IN

Autho r is atio nSe rvic e


C lie nt


File C atalo gue

File C atalo gue

D BD rive r

Authentication


C lie nt


File C atalo gue

File C atalo gue

D BD rive r

File transpo r tSe rvic e

D ISK

Data access

--./| |--r3418_01-01.ds| |--r3418_02-02.ds| |--r3418_03-03.ds| |--r3418_04-04.ds| |--r3418_05-05.ds| |--r3418_06-06.ds| |--r3418_07-07.ds| |--r3418_08-08.ds| |--r3418_09-09.ds| |--r3418_10-10.ds| |--r3418_11-11.ds| |--r3418_12-12.ds| |--r3418_13-13.ds| |--r3418_14-14.ds| |--r3418_15-15.ds

D0

pathdirhostIndexentryId

char(255)integer(11)integer(11)integer(11)

<fk><pk>

T2526

typedirnameownerctimecommentcontentmethodmethodArggownersize

char(4)integer(8)char(64)char(8)char(16)char(80)char(255)char(20)char(255)char(8)integer(11)

T2527

typedirnameownerctimecommentcontentmethodmethodArggownersize

char(4)integer(8)char(64)char(8)char(16)char(80)char(255)char(20)char(255)char(8)integer(11)

Bookkeeping


AliEn as a meta-GRID

AliEn User Interface

AliEn stackiVDGL stack EDG stack


DataGrid & PROOF

Local

Remote

Selection

Parameters

Procedure

Proc.C

Proc.C

Proc.C

Proc.C

Proc.C

PROOF

CPU

CPU

CPU

CPU

CPU

CPU

TagDB

RDB

DB1

DB4

DB5

DB6

DB3

DB2

Bring the KB to the PB and not the PB to the KB


Parallel Script Execution

root

Remote PROOF Cluster

proof

proof

proof

TNetFile

TFile

Local PC

$ root

ana.Cstdout/obj

node1

node2

node3

node4

$ root

root [0] .x ana.C

$ root

root [0] .x ana.C

root [1] gROOT->Proof(“remote”)

$ root

root [0] .x ana.C

root [1] gROOT->Proof(“remote”)

root [2] gProof->Exec(“.x ana.C”)

ana.C

proof

proof = slave server

proof

proof = master server

#proof.confslave node1slave node2slave node3slave node4

*.root

*.root

*.root

*.root


Conclusions: Not presented

Scripting languages (Python, CINT) GUI and graphics Detector Geometry Data Bases G3, G4, Fluka saga Conditions Data Bases Code Management tools on top of CVS Mathematical libraries Experiment Frameworks Progress expected

in these areasin the coming two years


Conclusions

Things converging rapidly in the past few months.

A common solution for I/O in view. Hybrid solution (Root-based event store +

RDBMS system(s) + GRID catalogs First GRID tools/systems appearing Discussions between major developers Still many problems to be solved.

Date post:	13-Dec-2015
Category:	Documents
Upload:	ferdinand-matthew-hardy
View:	214 times
Download:	1 times

INSTR021 Computing in High Energy Physics INSTR02 March 6 2002 Ren é Brun CERN.

Documents