ATLAS HLT/DAQ

ATLAS HLT/DAQ ATLAS HLT/DAQ

V. Vercesi for the ATLAS Italia HLT/DAQ GroupV. Vercesi for the ATLAS Italia HLT/DAQ Group

CSN1 Trieste Settembre 2006CSN1 Trieste Settembre 2006

CSN1 Trieste Settembre 2006 V. Vercesi - INFN Pavia 2

S. Falciano (Roma1) Coordinatore Commissioning HLT A. Negri (Irvine, Pavia) Coordinatore Event Filter Dataflow A. Nisati (Roma1) TDAQ Institute Board chair e Coordinatore PESA Muon Slice F. Parodi (Genova) Coordinatore b-tagging PESA V. Vercesi (Pavia) Deputy HLT leader e Coordinatore PESA (Physics and Event

Selection Architecture) Attività italiane

Trigger di Livello-1 muoni barrel (Napoli, Roma1, Roma2) Trigger di Livello-2 muoni (Pisa, Roma1) Trigger di Livello-2 pixel (Genova) Event Filter Dataflow (LNF, Pavia) Selection software steering (Genova) Event Filter Muoni (Lecce, Napoli, Pavia, Roma1) DAQ (LNF, Pavia, Roma1) DCS (Napoli, Roma1, Roma2) Monitoring (Cosenza, Napoli, Pavia, Pisa) HLT/DAQ system commissioning and exploitation (Everybody)


ATLAS TDAQATLAS TDAQ

Dual(quad)-CPU nodes

SDX1

USA15

UX15

ATLASdetector

Read-Out

Drivers(RODs) First-

leveltrigger

Read-OutSubsystems

(ROSs)

UX15

USA15

Dedicated links

Timing Trigger Control (TTC)

1600Read-OutLinks

Gig

abit

Eth

erne

t

RoIBuilder

Reg

ions

Of

Inte

rest

VME~150PCs

Data of events acceptedby first-level trigger

Eve

nt d

ata

requ

ests

Del

ete

com

man

ds

Req

uest

ed e

vent

dat

a

Event data pushed @ ≤ 100 kHz, 1600 fragments of ~ 1 kByte each

LVL2Super-visor

DataFlowManager

EventFilter(EF)

pROS

~ 500 ~1600

stores LVL2output

~100 ~30

Network switches

Event data pulled:partial events @ ≤ 100 kHz, full events @ ~ 3 kHz

Event rate ~ 200 HzData

storage

LocalStorage

SubFarmOutputs

(SFOs)

LVL2 farm

Network switches

EventBuilder

SubFarmInputs

(SFIs)

Second-leveltrigger


Efficienza dell’algoritmo di selezione è > 99%

Tutte le inefficienze sono dovute a zone dello spettrometro non coperte da RPC. In particolare: Settori speciali (Feet), ascensore, crack =0 e supporti del magnete.

Efficienza dell’algoritmo di selezione è > 99%

Tutte le inefficienze sono dovute a zone dello spettrometro non coperte da RPC. In particolare: Settori speciali (Feet), ascensore, crack =0 e supporti del magnete.

No RPC hitNo TriggerLow-Pt Trigger

DC3 data

DC2 data

Muon sources6 GeV6 GeV threshold

Lumi=1033

20 GeV20 GeV threshold

Lumi=1034

/K 9300 1090

b 1620 700

c 943 300

W 3 27

t Negligible Negligible

Low-pT 6 GeV Threshold

Efficienza globale Level-1 nel Barrel: Efficienza globale Level-1 nel Barrel: 83% Low-pT Trigger 79% High-pT Trigger83% Low-pT Trigger 79% High-pT Trigger

Level-1 Muon Trigger SystemLevel-1 Muon Trigger System


One Switch

rack-

TDAQ rack-

128-port GEth for L2+EB

One ROS rack

-

TC rack+ horiz. Cooling

-

12 ROS48 ROBINs

One Full L2

rack-

TDAQ rack-

30 HLT PCs

PartialSuperv’r

rack-

TDAQ rack

-3 HE PCs

Partial EFIO rack

-

TDAQ rack

-10 HE PC(6 SFI - 2 SFO - 2 DFM)

Partial EF rack

-

TDAQ rack

-12 HLT

PCs

Partial ONLINE

rack-

TDAQ rack-

4 HLT PC(monitoring)

2 LE PC(control)2 Central

FileServers

RoIB rack

-

TC rack + horiz. cooling

-50% of RoIB

5.5

surface: SDX1underground : USA15

Pre-series system in ATLAS point-1Pre-series system in ATLAS point-18 racks (10% of final dataflow, 2% of 8 racks (10% of final dataflow, 2% of EF)EF)

•ROS, L2, EFIO and EF racks: one Local File Server, one or more Local Switches

•Machine Park: Dual Opteron and Xeon nodes, uniprocessor ROS nodes

•Operating System: Net booted and diskless nodes, running SLC3


Event BuildingEvent Building

Parameterizing observed EB measurements,we better understand the conditions for stable& performant operations

TS = number ofrequests fromSFI nodes forevent fragments

WT = number ofevents processedin parallel by L2 farm

= L2 accept ratio


Pre-series studiesPre-series studies

L2 perf. scalability with muon algo

0

5

10

15

20

25

30

0 10 20 30 40 50

# L2PU nodes

thro

ughp

ut, k

Hz

1 L2SV, 1app/node

1 L2SV, 3apps/node

2 L2SVs, 1app/node

2 L2SVs, 3apps/node

Fraction of events processed by LVL2 as a function of the decision latency

0

0,2

0,4

0,6

0,8

1

1,2

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29

Latency (ms)

Fra

cti

on

of

ev

en

ts

mu

jet

e


ROD Crate DAQROD Crate DAQ Last major upgrades

To accommodate detectors’ requests New API for operational monitoring

To get “coherent” operation information Simultaneous module readout in a crate

RCD creates a packet of info and sends it to Monitoring A monitoring task analyses the packet

Further improvements in last TDAQ version OKS schema for RCD and ROS simplified Detector modules in a crate can be managed

Serially or in multi-threaded mode Each state transition can be set as multi-threaded

Can dramatically improve initialization time

RCD plans The RCD structure should be stable enough

All the main requests from detectors have been fulfilled We continue supporting the implementations

Main task is now RCD commissioning Goes on with ROD commissioning Includes support to all detectors


Monitoring Framework Monitoring Framework


MonitoringMonitoring

Reliable and stable GNAM CORE Configurable from on-line database, with histogram history Logging system (to migrate to ERS) Access Manager to ATLAS conditions DB

Detector libraries MDT, RPC, Tile, CSC, (TGC): in use for commissioning Interface to Event Display available (for some detectors)

Main issue: Smart Monitoring Automatically catch most low-level problems Needed for reaching and maintaining stable running conditions Common tools needed

Histogram comparison toolkit Error message routing and filtering

Detector experts Identify most common problem sources Implement and test alarm strategies


HLT Selection algorithmsHLT Selection algorithms

Lots of ongoing activities on algorithm development side LVL1 and LVL2 algorithms run in 11.0.6 CSC production for all trigger

slices EF algorithms almost finished (target 12.0.3, next week) Lots of ongoing work to improve/test/validate algorithms PESA algorithms being reviewed (Project Milestone)

ID LVL2, ID EF, Calo LVL2 e/g reviews have already taken place Muon LVL2+EF ~90% complete, Calo EF e/ in October Calo jet/tau/etmiss simple extension of e/ Building up comprehensive information about performance

Ongoing work for testing in athenaMT/athenaPT Measurements of system (timing) and physics performance LVL1/HLT AODs fully available in Rel 12 for trigger-aware

analyses (Project Milestone) Most data in place for ESD & AOD in release 11.0.5, Apr 06 More hypothesis algorithms, slice configs and TriggerDecision in 12.0.1


HLT SteeringHLT Steering

Il cuore delle sequenze di selezione e decisione Sviluppi basati sull’attuale implementazione (12.0.2)

Introduzione di selezioni topologiche elementari Primi test attualmente in sviluppo nel settore della B-fisica (B→, J/Psi→,

BS→DSconDS→) Introduzione di meccanismi per la sincronizzazione delle sequenze di

algoritmi Selezioni combinate di diversi oggetti (ad es. singolo e, singolo jet, singolo ) Sincronizzazione delle selezioni topologiche

Iterazione del design di implementazione Ridisegno dell’implementazione dello Steering

Nuove interfacce, più generali e flessibili, per gli algoritmi di feature extraction e hypothesis testing

Ristrutturazione della navigazione, disaccoppiata dallo Steering, per essere facilmente utilizzata nelle analisi Offline

Interfacciamento con l’introduzione del concetto di Trigger Chain Fase di disegno quasi ultimata: inizio dell’attività di sviluppo e test


HHighighLLighighT e/gammaT e/gamma

Electrons Eff. w.r.t truth for ET=25GeV e±

Time per track in top events (~1.3 tracks/RoI with pTmin=0.5GeV)

Brem recovery to be optimized: in progress…

Optimizations ongoing also for eff/rej

Photons Use only calorimeter information at L2 and EF Use CSC data, single photons ET = 20, 60 GeV,

filtered QCD di-jets with ET(hard) > 17 GeV

97%

98%

Algorithm KalmanFitter GSFTime/track 9.5 ms 1.24 s

Eff %

20i

Rate

220i

Eff %

60

Rate

60

L1 98.1 330 Hz 97.9 180 Hz

L2 Calo 92.7 6 Hz 94.4 49 Hz

EF Calo 85.3 4 Hz 84.3 42 Hz


Efficiencies for SM signalsEfficiencies for SM signals

trigger

2e15i 67.2% ---

e25i 92.9% 79.6%

e60 20.4% 6.9%

all 94.8% 80.3%

Z ee W e

Efficiencies given after kinematical cuts:• 2e in ||<2.5 with ET>15 GeV for Zee• 1e in ||<2.5 with ET>25 GeV for We

Use trigger optimisations giving 80% overall eff (outside crack) for 2e15i, e25i, e60.Use trigger optimisations giving 80% overall eff (outside crack) for 2e15i, e25i, e60.

e25ie25i

• efficiency stays flat after turn-on.efficiency stays flat after turn-on.• Turn on curves similar for e60.Turn on curves similar for e60.

# after L1+L2+EF# after L1+L2+EF# after kinematical cuts# after kinematical cuts

effeff

Method for single-electron trigger efficiency determination from data in Method for single-electron trigger efficiency determination from data in progress ( progress ( , ), )Z ee Z


Trigger efficiency from Z → μTrigger efficiency from Z → μ++μμ--

Double Object (DO) method Control sample: “Good” Z from 2

offline +- with loose selection cuts + 1± trigger signature satisfied

Trigger efficiency determined from counting in how many cases the second ± satisfies the trigger requirements

EF TrigMoore (MuonSpectrometer standalone)

EF TrigMoore (MuonSpectrometer + InnerDetector)

Reco MCεEF (%) 96.4 ± 0.2

εL1+EF (%) 79.9 ± 0.3 83.2 ± 0.2

Reco MCεEF (%) 94.5 ± 0.2

εL1+EF (%) 78.3 ± 0.3 80.2 ± 0.2

For (µ20)>70% statistical uncertainty after 30min at L=1033cm-2s-1: ~1-2%

Ongoing study using complementary method of orthogonal signature from ID (DOS method), almost finished


HHighighLLighighT muonsT muons Barrel:

Final update of the LUT available: optimized with respect to the momentum resolution and to the efficiency

Hypothesis exists and the PT cut is optimized for the 6 GeV threshold

Endcap: Bugs found in the TGC reconstruction

code and in the analysis programs Situation improved: harsh zones still

remains, but results are understood now Only TGC data has been used: try to

measure muon PTand to parameterize the track path

An offline study of the momentum resolution obtained fom MDT data is also under way: preliminary results are in agreement with those we got using only TGC data.

Large dishomogenity

Endcap stations outside magnetic field

Innermost TGC wheelshave reduced coveragein small sectors

EndCap

6 GeV 20 GeV

Endcap: 1.05 <ll <2.4

CSC data

prelim.

pT (GeV)

effic

ienc

y


HHighighLLighighT Jet/Tau/ETmissT Jet/Tau/ETmiss

Very active trigger slices, just one year ago only small part of the selections were available (new groups joining TDAQ)

Jets Algorithms for LVL1/LVL2/EF reco

and hypothesis in place Lots of work carried out to allow

trigger-aware analysis Ongoing work to understand

physics performance

Tau Algorithms for LVL1/LVL2/EF reco and hypothesis in place Use tauRec at EF, support for other tau reco packages foreseen Ongoing work to find best sequence at LVL2: first calo reconstruction or first

tracking reconstruction L2 selection ran successfully in athenaMT in 11.0.6

ETmiss Study performance of ETmiss at LVL1 and EF


Prescale JetsPrescale Jets

First place where to study and implement pre-scales

Reconstructed Jet ET

distribution for a combination of J1 to J8 samples, scaled according to their respective cross-section


Algorithms in Pre-SerieAlgorithms in Pre-Serie Motivation

Get the trigger PESA algorithms working online Make sure the latest offline developments get running online

Historical background Tools allowing the emulation of online running were developed

athenaMT (LVL2) and athenaPT (EF) A group of people was formed in order to provide full slice job options suitable for

online running Jets (Ignacio Aracena) Taus (Pilar Casado, Richard Soluk) Egamma (Xin Wu, Imma Riu) Muons (Alessandro Di Mattia, Diana Scannicchio)

One single integrated job options built successfully able to run in the Pre-Serie L2 and EF farms from real data format (bytestream)

Try this integrated job options in an online partition with different input files Single electrons, Top events, Muon events Produce a unified input BS file to be tested online

Produce a BS output file from SFO and try extending the menu to other signatures (e.g. di-objects) using the same algorithms slices to see the trigger behaviour


RunningRunning

L2 partition running ONLINE

single electrons withthe integrated job optionsin the nightlies of 11.0.6


More runningMore running


DatabasesDatabases

OnlineOracleDB

Offlinemaster

CondDB

Tier-0reconreplica

Diagnosticsreplica

Tier-1replica

Tier-1replica

OnlineDetectorsHLT farm

Diagnostics

Tier-0 farm

Bypass(ATCN /CERN-IT)

ATLAS Point1CERN Computer centre

Remote sites

Isolation / cut

Calibration Updatesincluding special Muons calibration path

ATCN Network

OKSin-memory

OODB

COOLRelDB

CORAL*

DCS

AthenaServer

Data: in the DB, plus in (large) files

referenced from the DB

RDB systems used: Oracle, MySQL, sqlite

*layer to decouple clients


Trigger ConfigurationTrigger Configuration TriggerTool

GUI for DB population easy and consistent menu changes for

experts (LVL1 and HLT) TriggerDB

stores all information to configure the trigger: LVL1 menu, HLT menu, HLT algorithm parameters (JO), HLT release information

stores all versions used, with a key Configuration and Conditions DB DB available at Point 1 and replicated

to external sites

Retrieval of information for running get information by key via two paths extraction of data in XML/JO files direct read-out

for both online + offline running

TriggerDB

onlinerunning

offlinerunning

shift crewoffline user expert

TriggerTool DB population scripts

Data Flow:

R/O interface ConfigurationSystem

compilers


Example database for TrigConfExample database for TrigConf

LVL1 HLT jobOptions

HLT

menu HLT

release

keys: stored in Conditions DB => Trigger conditions precisely known


HLT Large Scale TestsHLT Large Scale Tests Test complete HLT on many nodes

Scalability of DAQ, DB, HLT sw Emphasis on DB, HLT (no LVL1) DAQ: verify scalability & stability DataBase:

Explore caching tools (DBProxy) Detector oriented tests

1000 to 1200 nodes LST 2005: 600 (512 used for LVL2)

Will be provided by IT (agreed) with lxbatch etc. machines Non-optimized network

Big enough for full size Level2, ¼ size Event Filter Test Level-2 and Event Filter together (+EB and SFO)

As many trigger algorithms as possible Large number of events, recycled Collect monitoring info

Physics plots (produced by algorithms) Detailed timing histograms, etc.

Need to have SHIFTS!


LVL2 ID in cosmicsLVL2 ID in cosmics

Tracking algorithms IdScan, modified to handle

single tracks with large impact parameters

SiTrack, version with special LUT tuned for muons not coming from IP

Variables studied For T2Id, Offline and MC:

Ntracks, d0, phi0, z0, eta0 Efficiency calculations

Data: LVL2 wrt Offline Simulation: LVL2 and

Offline wrt truthCosmics at SR1:instrumented

region,two SCT sectors


Trigger-aware analysisTrigger-aware analysis Analyses using trigger

information as a “pre-processor” to correctly evaluate efficiencies, physics reach, etc.

The reconstructed objects, used by the trigger are saved in the ESD/AOD file

They can be used for comparison with truth/reconstructed information

It is possible to re-play the trigger decision, by running the hypothesis algorithms on these objects

Only the settings of the hypothesis algorithms can be changed in the analysis

The effect of different threshold settings can be measured

Production AnalysisData taking


Trigger & Physics WeeksTrigger & Physics Weeks Follow up on TDAQ and Computing operation

e.g. where are the bottlenecks on (instantaneous and average) rates? Updates on trigger slices and initial feedback on Trigger Aware Analyses in

Release 12 Work towards ultimate offline efficiencies and rejections, to be compared to

corresponding figures for trigger selections Especially relevant for Combined Performance e, t, m, b-tag

Follow up on minimum-bias event selection and analysis Follow up on menu for L = 1 × 1031 cm-2s-1 (items and rates)

Small group formed to study the issue and propose a strategy Ideas shown need to be developed into menus for commissioning and for early

physics (optimisation for two purposes somewhat different) Follow up on physics with b-jets

Efficiency to identify the jets for tagging in LVL2 in various physics scenarios - e.g. rates and RoI multiplicities

“Hot topics” End-cap LVL2 muons Forward-jet trigger ETmiss rates (including beam backgrounds)

Next week end of October


Planning…Planning…

Schedula in revisione alla luce delle nuove informazioni su LHC:

discussioni durante la TDAQ Week di Settembre:

cioè adesso, in perfetta sovrapposizione con la nostra CSN1

(Ho il volo per Londra domani mattina)


ROS planROS plan

Prima tranche dei ROS già pagata al CERN (275 KCHF) Seconda tranche e resto delle network card ordinata e consegna ultimata di

recente, pagamento previsto entro la fine dell’anno (275 KCHF) Questo esaurisce il nostro contributo CORE al Read-Out System

Impegno importante previsto in ME per contributi all’installazione e commissioning dell’intero sistema Sezioni coinvolte Pavia, Roma1, …


Data Collection planData Collection plan

Other DataFlow PCs Price enquiry finished for SFI, DFM, L2SV and pROS Evaluations nearly complete

Choose which company(ies) to buy from SFOs - specification to be finalised in ~1 month

2007 CORE expected from Italy: 50 KCHF Try and setup system capable of deliverying steady-state nominal SFO output from day 1

Better position to exploit early running trigger and detector studies No correlation with total resources available in HLT farms

e.g. send LVL1 output directly to off-line


Networks and Online planNetworks and Online plan

Switch Procurements (Chassis + Pizza Boxes) Plan finalised, shopping list produced for 2006 Sharing between FA’s agreed

File servers and Monitoring PC specs finalised 135 KHCF su CORE 2006, inclusi monitoring PC e local file server: end of contributions


Infrastructure planInfrastructure plan

SDX Infrastructure Orders started - cable ladders, nuts & bolts, shelves for switches About to decide on power distribution

Coolers for SDX Order placed for 43 racks (all of upper level)

2007 CORE expected from Italy: 80 KCHF


HLT farms planHLT farms plan

HLT Processors IT Market Survey completed Specifications has been drafted: similar (but not identical) to ATLAS needs Not possible to use a blanket contract for 2006: buy our own

Lively discussion about spending profile for this items First iteration ended Saturday September 16th @ 3:44 (Big Ben time) Propose today our best approach based on

Need to cover basic needs for commissioning, calibrations, first run period Avoid buying anything that is not absolutely necessary 95 KCHF back to CSN1 in 2006

Other HLT/DAQ Items First bills are appearing here - Rack coolers, inter-rack fibres, etc

TDAQResource Committee


Revised Cost Profile (KCHF)Revised Cost Profile (KCHF)

2004 2005 2006 2007 2008 2009 Total

Pre-series 140 0 0 0 0 0 140

Detector R/O 0 275 275 0 0 0 550

LVL2 Proc 0 0 30 95 365 160 650

Event Builder 0 0 50 50 110 70 280

Event Filter 0 0 110 140 670 380 1300

Online 0 45 135 0 0 0 180

Infrastructure 0 0 80 80 20 20 200

INFN Total 140 320 680 365 1165 630 3300

TDR Total 1048 3357 4087 4544 7522 4543 25101


Richieste ME (DAQ/HLT)Richieste ME (DAQ/HLT)

Per le responsabilità vedi la prima trasparenza Installazione e commissioning ROS e HLT

4 mesi uomo a Roma1 2 mesi uomo a Pavia

Commissioning DAQ muoni/pixel 4 mesi uomo a Roma1 (incluso ROD Crate DAQ) 2 mesi uomo a Pavia (incluso sviluppo Monitoring) 9 mesi uomo a Genova (incluso HLT steering)

Event Building, SFI e SFO 8 mesi uomo a LNF (inclusi Large Scale Tests)

Deployment degli algoritmi di selezione on-line 2 mesi uomo a Pavia


ConclusioniConclusioni Stato attuale del progetto HLT/DAQ ben allineato con le scadenze future di

ATLAS e di LHC previste nel 2007 Il progetto è certamente complesso e anche le responsabilità e i finanziamenti

italiani coprono diversi settori Il sistema di HLT/DAQ ha legami profondi con tutte le altre aree di sviluppo:

rivelatori, software online e offline, networking, performance di fisica, etc Sono necessari maggiori contributi alla forza lavoro per la parte di installazione e

test Impegno fondamentale di consentire a tutti i rivelatori una fase efficiente di

commissioning Inserimento degli algoritmi nei test di cosmici Realizzazione di catene complete di read-out e successivamente di Event Building Le slice HLT hanno raggiunto uno stadio di maturità avazata, sono integrate

nell’ambiente on-line e costituiscono gli ingredienti per la costruzione e lo studio dei Menu di Trigger

Il Progetto è in fase di revisione (management) Necessità di adatttarsi alla nuova fase dell’esperimento Coerenza con l’ATLAS Operational Model Impegno perché gli italiani mantengano i ruoli di visibilità che si sono meritati

Date post:	09-Jan-2016
Category:	Documents
Upload:	rayya
View:	107 times
Download:	2 times

ATLAS HLT/DAQ

Documents