1

The MEG Software The MEG Software ProjectProject

PSI 9/2/2005Corrado Gatto

Offline Architecture

Computing Model

Status of the Software

Organization

2

The Offline ArchitectureThe Offline Architecture

3

Dataflow and Reconstruction Dataflow and Reconstruction RequirementsRequirements

100 Hz L3 triggerevt size : 1.2 MBRaw Data throughput:

(10+10)Hz1.2Mb/Phys evt 0.1 + 80Hz0.01Mb/bkg evt =

3.5Mb/s <evt size> : 35 kBTotal raw data storage: 3.5Mb/s 107s = 35 TB/yr

4

Requirements for Software ArchitectureRequirements for Software Architecture

Geant3 compatible (at least at the beginning)

Easy interface with existing packages:

– Geant3 , Geant4, external (fortran) event generators Scalability

Simple structure to be used by non-computing experts

Written and maintained by few people

Portability

Use a world-wide accepted framework

Use ROOT + An existing Offline Package as starting point

5

Expected Raw PerformanceExpected Raw Performance

0.0

20.0

40.0

60.0

80.0

100.0

120.0

140.0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Number of event builders

Glo

bal t

hrou

ghpu

t (M

B/s)

Pure Linux setup20 data sourcesFastEthernet local connection

6

MC & Data Processing SchemeMC & Data Processing SchemeSimu sig Simu bkg 1

hits files sig hits files bkg 1

SDigitizer

SDigits files sig

SDigitizer

SDigits files bkg

Digitizer

Merged Digits filesRaw Data

Reconstruct

ESD

Reconstruct

ESD

C++

Fortran & G3Or

C++ & G3/G4/Fluka

Simu bkg 2

hits files bkg 2

SDigitizer

SDigits files bkg Fortran & G3Or

C++

7

General Architecture: GuidelinesGeneral Architecture: Guidelines Ensure high level of modularity (for easy of maintenance)

Absence of code dependencies between different detector modules (to C++ header problems)

The structure of every detector package is designed so that static parameters (like geometry and detector response parameters) are stored in distinct objects

The data structure is build up as ROOT TTree-objects

Access is possible to either the full set of correlated data (i.e., the event) or only one or more sub-sample, stored in different branches of the data structure (TBranch class) and corresponding to one or more detector.

8

Computing ModelComputing Model

9

Elements of a Computing ModelElements of a Computing ModelComponents

– Data Model Event data sizes, formats,

streaming Data “Tiers” (DST/ESD/AOD

etc) Roles, accessibility,

distribution,… Calibration/Conditions data Flow, latencies, update freq Simulation, Sizes, distribution File size

– Analysis Model Canonical group needs in

terms of data, streams, re-processing, calibrations

Data Movement, Job Movement, Priority management

Interactive analysis

Implementation– Computing Strategy and

Deployment Roles of Computing Tiers Data Distribution between Tiers Data Management Architecture Databases Masters, Updates, Hierarchy Active/Passive Experiment

Policy– Computing Specifications

Profiles (Tier N & Time)– Processors, – Storage, – Network (Wide/Local),– DataBase services,– Specialized servers

Middleware requirements

10

CPU NeededCPU Needed

Calibration: 1 - 7 CPU’s MC Production: 11 - 33 CPU’s MC Reconstruction: 3 - 14 CPU/repr. Raw data reconstruction: 3 – 14 CPU/repr. Alignment: to be estimated

Assume:– Trigger rate: 20 Hz– Beam Duty Cycle: 50%– MC = data– Calibration: 1Hz

Estimates by scaling existing code and a C++ framework

Optimal solution for reconstruction:take Beam Duty Cycle: 100% and use no-beam time for reprocessing. -> Double the CPU for reconstruction.

11

Storage Needed Storage Needed Assume:

– Raw data (compressed) event size: 120 kB– Hits (from MC): 9 kB/evt– Sdigit+Digit size (from MC): 30kB + 120kB– Track reference (from MC): 15 kByte– Kinematics (from MC): 4 kByte– ESD size (data or MC): 30 kByte

Storage required yearly (L3 trigger: 20Hz and DC: 50%):– Raw data: 12 Tbyte (+ calib)– Hits (from MC): 0.9 TByte– Digit size (from MC): 15 TByte– Track reference (from MC): 1.5 TByte– Kinematics (from MC): 0.4 TByte – ESD size (data or MC): 3/Tbyte /reproc.

From Alice data model

12

A ComputingA Computing ModelModel for MEG for MEG

The crucial decisions are being taken at the collaboration level

Very dependent on CPU and storage requirement

Two degrees of complexity are being consideredINFN has requested a CM design by Nov 2005.

13

Data Access: ROOT + RDBMS ModelData Access: ROOT + RDBMS Model

histograms

Calibrations

Geometries

Run/FileCatalog

Trees

Event Store

ROOTfiles

OracleMySQL

14

Computing Deployment: Computing Deployment: Implementation #1Implementation #1

Concentrated computing– Will be considered only in the case the CPU’s

needed would exceed PSI’s capabilities– Easiest implementation– It requires the least manpower for maintenance– PROOF is essential for Analysis and DQM

15

Computing Deployment: Computing Deployment: Implementation #2Implementation #2

Distributed computing– Posting of the data/Catalog

synchronization– Data distribution dependent

upon connection speed– PSI -> INFN OK (3 MB/sec)– PSI -> Japan needs GRID

Department

Desktop

PSI – Tier 0(PSI - Tier 1)

Tier 1 X Y

Z

622 Mbps

2.5 Gbps

622 M

bp

s

155

mbp

s

155 mbps

Tier2 Lab aUni bLab c

Uni n

"Transparent" user

access to applications and all data

16

Processing FlowProcessing Flow

An

a

DPD

Raw

CalibC

alib

&R

eco

1

ESD

AOD

Tag

R

eco

Rep

roc

MC

ESD

AOD

Tag

17

Computing Model ConfigurationComputing Model Configuration

Multi-T ier S tructure

A n a lys is 1

T ie r 1P isa

M C P rod u c tionR e p roce ss.

A n a lys is 2

T ie r 1L e cce

M C P rod u c tionR e p roce ss.

A n a lys is 3

T ie r 1T o kyo

M C P rod u c tionR e p roce ss.

A n a lys is 4

T ie r 1P S I

M C P rod u c tionR e p roce ss.

T ie r-0P S I

P ro m pt C a lib .E v t R e co

18

How We Will ProceedHow We Will Proceed1. Decide for single-Tier or multi-Tier CM (depends heavily on

CPU+Storage needed)2. PSI is OK for Tier-0 (existing infrastructure + Horizon Cluster)3. Find the candidate sites for Tier-1

Requirement for a Tier-1Requirement for a Tier-11. 1 FTE for job running, control2. 0.5-1 FTE for data export, catalog maintenance and DQM3. If GRID is needed, 0.3-0.5 FTE for GRID maintenance4. Software installation would be responsibility of the Offline

group (after startup)

19

Computing Model StatusComputing Model Status Started interaction with PSI’s Computing Center (J. Baschnagel

and coll.) PSI CC survey was very encouraging: Physical space and electrical power not a limitation However, CC is running near 80% of Cooling and Clean Power

capabilities Relevant services (backup, connection to exp. Area) are OK Alternative options are being considered (setup the farm in the

experimental area and use the CC only for backup) Horizon Cluster PSI stand well as Tier-0 candidate (in a MultiTier CM. It might have enough resources to fulfill all MEG’s computing

requirement

20

Status of the Status of the SoftwareSoftware

21

Software OrganizationSoftware Organization Development of Montecarlo (G3+Fortran) and

Calibration/Reconstruction (VMC+ROOT) will initially proceed in parallel

Migration will eventually occur later (end 2005) Montecarlo group is in charge of the simulation

(Geometry+Event Generator) Offline group in charge of the rest (architecture, framework,

reconstruction, computing, etc.) Detector specific code developed along with Detector Experts Present analyses for R&D within the ROME environment (same

as Online)

22

Status of the Montecarlo: DCHStatus of the Montecarlo: DCH(P. Cattaneo)(P. Cattaneo)

Progress– Implementation of cable duct and various mechanical supports– Reorganization of the GEANT volume inside the chambers– First integration of the time to distance profile from GARFIELD

Next– Completing integration of GARFIELD profiles– Calculation of time profiles for signals– Effect on signals of electrode patterns– Electronic simulation– Signal digitization– Improving details cards, cables

23

Status of the Montecarlo: EMCALStatus of the Montecarlo: EMCAL Progress

– Pisa model: implement new geometry– Pisa model: new tracking model (not based on GTNEXT)– Tokio model: GEANT based (GNEXT) photon tracking including

Refraction PMT quartz window Refraction PMT holder Tokio model: absorption Tokio model: scattering

– Tokio model: ZEBRA output Number of p.e. in each PMT PMT position (to be replaced by MySQL/ XML)

– Tokio model: ZEBRA2NTuple converter

Next– Update geometry and complete integration between Pisa and Tokio– Signal digitization– ZEBRA output for hit timing

24

Status of the Montecarlo: TOFStatus of the Montecarlo: TOF Progress

– Implementation of the actual geometry: tapered slanted bar, square fibers

– Addition of phototubes, photodiodes and related mechanics

Next– Generation of photons in the bars/fiber– Propagation of photons to the PMT & photodiodes– Electronics and PMT/ photodiodes simulation– Improving details of material distribution, e.g. better PMT,– cables– Add mechanical support– Signal digitization

25

Status of the Montecarlo: moreStatus of the Montecarlo: more Graphics display of geometry only (no event)

– Some zooming capability– Possibility of displaying a single volume– Addition of phototubes, photodiodes and related mechanics

Beam, Target and Trigger– Preliminary approach outside GEM

26

Softwares used in LP beam testSoftwares used in LP beam test(R. Sawada)(R. Sawada)

Midas

Data taking

Slow control

Data logging

Run control

ROME + ROOTOnline monitoring

Offline analysis

MySQL

Channel information

Geometry information

Calibration constants

Proceedure of data takeProceedure of data take

BoRBoR EoREoREventEvent

run numbertime

trigger modeConfiguration ID

trigger modechannel info.

geometry info.

calibrationdata

data data

HistogramTree

(Ntuple)

time# of events

calibration

Proceedure of Offline AnalysisProceedure of Offline Analysis

BoRBoR EoREoREventEvent

trigger modechannel info.

geometry info.calibration

data

processeddata

calibration

Bofore a RunBofore a Run

calibration run number

29

Status of the Offline ProjectStatus of the Offline Project

Preliminary Offline architecture approved by MEG (June 2004)

INFN – CSN1 also approved the Project (Sept. 2004) Funds have been provided for installing a minifarm in

Lecce (22 kEUR) Real work started November 2004

30

Immediate Tasks and Objectives Immediate Tasks and Objectives Setup a system to test the Offline code

DAQ

Prompt

Calib

Reco farm

Online Disk

Server

Staging to tape

Prototype the Main Program Modules Classes Steering program FastMC

Test functionalities

& performance

Core Offline System

Development

RDBMS

31

Manpower EstimateManpower Estimate

2005Lecce 8.2Tokyo 1.0Rome 1.0Pisa 1.5Pavia 0.4Total 12.1

Available (FTE)Available (FTE)

Activity Profile 2005 2006 2007Off-line Coordination PH 0.8 0.8 0.8Off-line Framework Develop. PH/CE 1.2 1.2 1.2Collaborating Class Develop. PH 2.0 2.0Databases (Design & Maint) CE 1.0 0.5 0.5QA & Documentation PH/CE 0.5 0.5 1.0FastMC PH 1.0 1.0Montecarlo (hits+digi) PH 2.0 2.0 1.0Multiprocess. Develop. CE/PH 1.0 1.0 1.0I/O Development CE/PH 0.5 0.5MDC CE 0.5 0.5Syst. + web support CT 0.0 1.0 2.0Physics Tools 0.5 1.0 1.0Production PH/CE 2.0Total Needed 11.0 12.0 10.5

•+1 waiting

for funds

32

The People InvolvedThe People Involved Core Offline:

– Coordinator: Gatto– Steering Program: Di

Benedetto, Gatto– FastMC: Mazzacane, Tassielli– TS Implementation: Chiri– Interface to raw data: Chiri, Di

Benedetto, Tassielli– Calibration Classes & Interface

to RDBMS: Barbareschi*– Trigger: Siragusa

*Will start on Mar. 2005

Detector experts:– LXe: Signorelli, Yamada,

Savada– TS: Schneebeli (hit), Hajime

(Pattern), Lecce (Pattern)– TOF: Pavia/Genova– Trigger: Nicolo’ (Pisa)– Magnet: Ootani

•All <100%

•All 100% (except for Chiri)

Montecarlo:– LXe: Cattaneo, Cei, Yamada

•All <100%

33

MilestonesMilestonesOffline

1. Start-up: October 2004

2. Start deploying the minifarm: January 2005

3. First working version of the framework (FastMC + Reconstruction): April 2005 (badly needed to proceed with the Computing Model)

4. Initial estimate of CPU/storage needed: June 2005

5. Computing Model: November 2005

6. MDC: 4th quarter 2005

Montecarlo1. Include the calorimeter in gem

2. Keep the existing MC in the Geant3 framework. Form a panel to decide if and how to migrate to ROOT: 4th quarter 2005

34

MilestonesMilestonesOffline

1. Start-up: October 2004

2. Start deploying the minifarm: January 2005

3. First working version of the framework (FastMC + Reconstruction): April 2005 (badly needed to proceed with the Computing Model)

4. Initial estimate of CPU/storage needed: June 2005

5. Computing Model: November 2005

6. MDC: 4th quarter 2005

Montecarlo1. Include the calorimeter in gem

2. Keep the existing MC in the Geant3 framework. Form a panel to decide if and how to migrate to ROOT: 4th quarter 2005

35

ConclusionsConclusionsOffline project approved by MEG and the INFNOffline group has consolidated (mostly in

Lecce)Work has startedMinifarm for testing the software is being setupDefinition of the Computing Model is under wayMontecarlo is on its way to a detailed

description of the detector

36

Backup SlidesBackup Slides

37

Compare to the othersCompare to the others

38

Data Model (Monarc)Data Model (Monarc) ESD (Event Summary Data)

– contain the reconstructed tracks (for example, track pt, particle Id, pseudorapidity and phi, and the like), the covariance matrix of the tacks, the list of track segments making a track etc…

– AOD (Analysis Object Data)

– contain information on the event that will facilitate the analysis (for example, centrality, multiplicity, number of positrons, number of EM showers, and the like).

Tag objects– identify the event by its physics signature (for example, a cosmic ray) and is much smaller

than the other objects. Tag data would likely be stored into a database and be used as the source for the event selection.

DPD ( Derived Physics Data) – are constructed from the physics analysis of AOD and Tag objects. – They will be specific to the selected type of physics analysis (ex: mu->e gamma)– Typically consist of histograms or nt-uple-like objects. – These objects will in general be stored locally on the workstation performing the analysis,

thus not add any constraint to the overall data-storage resources

39

Recostruction StructureRecostruction Structure

Detector Class

Global Reco

Detectortasks

Run Class

Run Manager

Detector Class

DCH

Detectortasks

ROOT Data Base

TreeBranches

One or moreGlobal Objects

execute a list of tasksinvolving objects

from several detectors

Detector Class

EMC

Detectortasks

Run Class

MC Run Manager

Detector Class

TOF

Detectortasks

The Run managerexecutes

the detector objectsin the order of the list

Each detectorexecutes a list

of detector tasks

On demand actionsare possible butnot the default

40

Responsibilities & Tasks (all Responsibilities & Tasks (all software)software)

Offline Framework Develop. PSI (Schneebeli) + LecceClass Development Lecce + Detector expertsDatabases (Design & Maint) Tokyo (Savada)Computing Organization & Support LecceSimulation Tokyio (Yamada) + Pisa (Cei)Online Interface PSI (Ritt)Event Display & UI PSI + LecceSyst. + web support PSI + LecceProduction To be decided

Detector experts:– LXe: Signorelli, Yamada, Savada– DC: Schneebeli (hit), Hajime (Pattern), Lecce

(Pattern)– TC: Pavia/Genova– Trigger: Nicolo’ (Pisa)– Magnet: Ootani

41

Manpower Estimate (framework only)Manpower Estimate (framework only)

Profile Jul-04 2004 (4mo) 2005PH 0.2 1.2 1.2PostDoc 1.0Ph.D 1 2.0 3.0Fellowship 1.0 1.0Student 2.0 2.0CT 1.0Total 1.2 6.2 9.2

Available at LecceAvailable at Lecce•Job

posted: apply now

Activity Profile 2004 (4mo) 2005 2006 2007Off-line Coordination PH 0.8 0.8 0.8 0.8Off-line Framework Develop. PH/CE 1.2 1.2 1.2 1.2Collaborating Class Develop. PH 1.0 1.0 1.0Databases (Design & Maint) CE 1.0 0.5 0.5QA & Documentation PH/CE 0.5 0.5 1.0FastMC PH 1.0 1.0 1.0 1.0Multiprocess. Develop. CE/PH 1.0 1.0 1.0I/O Development CE/PH 0.0 0.5 0.5MDC CE 0.5 0.5Event Display & UI CE 1.0 1.0 0.5Syst. + web support CT 0.0 1.0 2.0Physics Tools 0.5 1.0 1.0Production PH/CE 1.0Total Needed 4.0 9.0 10.0 10.0

•4 from Naples + 2.2 from Lecce

•+1 waiting

for funds

42