One scenario forOne scenario forthe CBM Trigger the CBM Trigger
ArchitectureArchitecture
Ivan KiselIvan Kisel
Kirchhoff-Institut für PhysikKirchhoff-Institut für Physik, Uni-Heidelberg, Uni-Heidelberg
FutureDAQ Workshop, MünchenMarch 25-26, 2004
KIPKIP
25-26 March 2004, München25-26 March 2004, München Ivan Kisel, KIP, Uni-HeidelbergIvan Kisel, KIP, Uni-Heidelberg 22
Particle Multiplicities and Data RatesParticle Multiplicities and Data Ratesnchar
nneut
980
1080
680
700
1000
700
STS
60
Sub-System
GByte/secRICH
90
TRD
130
RPC
10
ECAL
50Total
350
Total
350
Multiplicities
for central
collision
Data rate for
107 int / sec
for Au+Au
25 A GeV
UrQMD
not including:
background
link overhead
W.Müller
25-26 March 2004, München25-26 March 2004, München Ivan Kisel, KIP, Uni-HeidelbergIvan Kisel, KIP, Uni-Heidelberg 33
DAQ-Trigger ScenarioDAQ-Trigger Scenario
Bandwidth 1 TByte/sec
Hit Level Processing
Local Level Processing
Buffering; Event association;
Regional Level Pre-Processing
3-10 MHz Event Rate
First Level Trigger:
FGPA; DSP; PC
300 kHz Event Rate
Second Level Trigger:
PC Farm
From
LoI
20 kHz Event Rate
1 GByte/sec Bandwidth
W.Müller
25-26 March 2004, München25-26 March 2004, München Ivan Kisel, KIP, Uni-HeidelbergIvan Kisel, KIP, Uni-Heidelberg 44
Modular Structure of DAQModular Structure of DAQ
MAPS, STS RICH ECALDetectorDetector
Trigger/Offline PC FarmTrigger/Offline PC Farm
Time-Slice Builder NetworkTime-Slice Builder Network
101077 ev/s ev/s
101055 slsl/s/s
50 50 kBkB//evev
100 100 evev//sliceslice
5 M5 MBB//sliceslice
N x MN x MN x MN x MSchedulerSchedulerSchedulerScheduler
TRD
Sub-FarmSub-Farm
RURURURURURURURURURURURURURURURU
RURURURURURURURURURURURURURURURU
Sub-FarmSub-Farm Sub-FarmSub-Farm Sub-FarmSub-Farm Sub-FarmSub-FarmFarm
Control SystemSub-FarmSub-FarmSub-FarmSub-Farm Sub-FarmSub-Farm Sub-FarmSub-Farm Sub-FarmSub-Farm
Sub-FarmSub-FarmSub-FarmSub-Farm Sub-FarmSub-Farm Sub-FarmSub-Farm Sub-FarmSub-Farm
Sub-FarmSub-FarmSub-FarmSub-Farm Sub-FarmSub-Farm Sub-FarmSub-Farm Sub-FarmSub-Farm
Sub-FarmSub-FarmSub-FarmSub-Farm Sub-FarmSub-Farm Sub-FarmSub-Farm Sub-FarmSub-Farm
SF
n
availab
le
SFnt MAPS STS RICH TRD ECAL
SFnt MAPS STS RICH TRD ECAL
SFntSFnt SFnt SFnt
25-26 March 2004, München25-26 March 2004, München Ivan Kisel, KIP, Uni-HeidelbergIvan Kisel, KIP, Uni-Heidelberg 55
Scheduled Data TransferScheduled Data Transfer
Deyan Atanasov, Scheduled Data Transfer, CBM DAQ, HD 18.03.04
The Scheduler assigns The Scheduler assigns Time-Slices to Sub-Time-Slices to Sub-Farms.Farms.
TXTX
EventsEvents
SourceSource
DestinationsDestinationsIN
Scheduler Scheduler CoreCore
SchedulingSchedulingDisciplineDiscipline
IN
OUT
1 2 3New
EventEntry
1. Idle.2. Obtain Destination.3. Produce a Tag.
25-26 March 2004, München25-26 March 2004, München Ivan Kisel, KIP, Uni-HeidelbergIvan Kisel, KIP, Uni-Heidelberg 66
Control
Experiment Control System
Monitor FCS Network
HostPCI bus
Farm Control System
FCS - Farm Control SystemFCS - Farm Control System
Cluster Interface Agent (CIA)• every node can be (re)configured, turned on/off remotely via Ethernet• it can save hardware components like display cards, Floppy Disk Drives• monitor the whole cluster, every host regardless of its current state with additional monitoring possibilities independent from operating system
•The farm implements its own The farm implements its own control networkcontrol network•Control over every node is Control over every node is done via a redundant done via a redundant hardware unit attached to the hardware unit attached to the nodenode•The interface between FCS The interface between FCS and ECS is done via a and ECS is done via a dedicated nodededicated node
Ralf Panse, Farm Control System, CBM DAQ, HD 18.03.04
25-26 March 2004, München25-26 March 2004, München Ivan Kisel, KIP, Uni-HeidelbergIvan Kisel, KIP, Uni-Heidelberg 77
Distributed Local Mass StorageDistributed Local Mass Storage
10 TB/PC10 TB/PC = 10 000 000 MB/PC = 10 000 000 sec = 120 = 10 000 000 MB/PC = 10 000 000 sec = 120 daysdays = =
4 4 months of data takingmonths of data taking
1 MB/sec/PC1 MB/sec/PC
Data TakingData Taking3 Months3 Months
Data AnalysisData Analysis9 Months9 Months
Data ReductionData Reduction1000:11000:1
Data ReductionData Reduction100:1100:1
Lord Hess, ClusterRAID1 Prototype, CBM DAQ, HD 18.03.04
Arne Wiebalck, ClusterRAID, CBM DAQ, HD 18.03.04
25-26 March 2004, München25-26 March 2004, München Ivan Kisel, KIP, Uni-HeidelbergIvan Kisel, KIP, Uni-Heidelberg 88
PC Sub-FarmPC Sub-Farm
Input Input DataData
Farm Control System
PCPCPCPC PCPCPCPCPCPCPCPCPCPCPCPC PCPCPCPC PCPCPCPC PCPCPCPCPCPCPCPCPCPCPCPC PCPCPCPC
FP
GA
FP
GA
FP
GA
FP
GA
FP
GA
FP
GA
FP
GA
FP
GA
PCPCPCPC PCPCPCPCPCPCPCPCPCPCPCPC PCPCPCPCPCPCPCPC PCPCPCPC
Sub-FarmSub-Farm Sub-FarmSub-FarmSub-FarmSub-Farm-Farm-Farm
•Various CPU powerVarious CPU power•Shared PCsShared PCs•ReconfigurableReconfigurable•Fault tolerantFault tolerant•Offline on backgroundOffline on background
FP
GA
FP
GA
FP
GA
FP
GA
FP
GA
FP
GA
FP
GA
FP
GA
FP
GA
FP
GA
FP
GA
FP
GA
FP
GA
FP
GA
FP
GA
FP
GA
SchedulerSchedulerSchedulerScheduler
25-26 March 2004, München25-26 March 2004, München Ivan Kisel, KIP, Uni-HeidelbergIvan Kisel, KIP, Uni-Heidelberg 99
MA
PS
MA
PS
STSSTS RICHRICH TRDTRDECALECAL
Local time slice 100Local time slice 100Local time slice 99Local time slice 99
Local time slice 1Local time slice 1Local time slice 2Local time slice 2Local time slice 3Local time slice 3
Local time slice …Local time slice …
Local time slice iLocal time slice iLocal time slice iLocal time slice i
MA
PS
MA
PS
STSSTS RICHRICH TRDTRD ECALECAL
LM/LM/DSPDSP LM/DSPLM/DSP LM/DSPLM/DSPLM/DSPLM/DSP
FPGA: Pre-process/L1FPGA: Pre-process/L1
NICNICNICNIC
InIn
NetworkNetworkNetworkNetwork
50
0 M
B/s
/FP
GA
50
0 M
B/s
/FP
GA
Fre
eFre
e// B
usy
Bu
sy
Offl
ine
Offl
ine
Local time slice i+1Local time slice i+1Local time slice i+1Local time slice i+1LM/LM/DSPDSP LM/DSPLM/DSP LM/DSPLM/DSPLM/DSPLM/DSP
FPGAFPGAPre-processPre-process/L1/L1 10000 ev/s10000 ev/s
100 100 s/evs/ev
Ou
tO
ut
PCsPCsPCsPCs
50
0 M
B/s
/FP
GA
50
0 M
B/s
/FP
GA
Memory BuffersMemory Buffers
25-26 March 2004, München25-26 March 2004, München Ivan Kisel, KIP, Uni-HeidelbergIvan Kisel, KIP, Uni-Heidelberg 1010
MA
PS
MA
PS
STSSTS RICHRICH TRDTRDECALECAL
Updated local time slice 100Updated local time slice 100
Updated local time slice 99Updated local time slice 99
Updated local time slice 1Updated local time slice 1Updated local time slice 2Updated local time slice 2Updated local time slice 3Updated local time slice 3
Updated local time slice …Updated local time slice …
CPU: L2/OfflineCPU: L2/Offline
NICNICNICNIC
InIn
FPGAsFPGAsFPGAsFPGAs
50
0 M
B/s
/CP
U5
00
MB
/s/C
PU
Fre
eFre
e// B
usy
Bu
sy
Offl
ine
Offl
ine
CPUCPUL2/OfflineL2/Offline
300 ev/s300 ev/s3 ms/ev3 ms/ev
StorageStorageStorageStorage
1 M
B/s
/CP
U1
MB
/s/C
PU
Memory BuffersMemory Buffers
25-26 March 2004, München25-26 March 2004, München Ivan Kisel, KIP, Uni-HeidelbergIvan Kisel, KIP, Uni-Heidelberg 1111
ECALECALECALECAL
FPGA 4D Pre-processor/L1 TriggerFPGA 4D Pre-processor/L1 Trigger
Reconstruct primary vertexSelect detached secondary D tracksSelect RoIs of secondary J/ tracksFit secondary tracksFit secondary verticesPre-process/Trigger
TRDTRDTRDTRDRICHRICHRICHRICH
STSSTSSTSSTS
STSSTSSTSSTS
STSSTSSTSSTS
STSSTSSTSSTS
STSSTSSTSSTS
25-26 March 2004, München25-26 March 2004, München Ivan Kisel, KIP, Uni-HeidelbergIvan Kisel, KIP, Uni-Heidelberg 1212
ECALECALECALECAL
ECALECALECALECALTRDTRDTRDTRD
RICHRICHRICHRICH
CPU 4D L2 TriggerCPU 4D L2 Trigger
Reconstruct primary vertexSelect detached secondary D tracksSelect RoIs of secondary J/ tracksFit secondary tracksFit secondary verticesTrigger
TRDTRDTRDTRDRICHRICHRICHRICH
STSSTSSTSSTS
STSSTSSTSSTS
STSSTSSTSSTS
STSSTSSTSSTS
STSSTSSTSSTS
Improve (m)any of the L1 results:Improve (m)any of the L1 results:
25-26 March 2004, München25-26 March 2004, München Ivan Kisel, KIP, Uni-HeidelbergIvan Kisel, KIP, Uni-Heidelberg 1313
ECALECALECALECAL
ECALECALECALECALTRDTRDTRDTRD
RICHRICHRICHRICH
ECALECALECALECALTRDTRDTRDTRD
RICHRICHRICHRICH
Offline AnalysisOffline Analysis
Reconstruct primary vertexSelect detached secondary D tracksSelect RoIs of secondary J/ tracksFit secondary tracksFit secondary verticesAnalysis
TRDTRDTRDTRDRICHRICHRICHRICH
STSSTSSTSSTS
STSSTSSTSSTS
STSSTSSTSSTS
Improve (m)any of the L2 results:Improve (m)any of the L2 results:
ECALECALECALECALTRDTRDTRDTRD
RICHRICHRICHRICH
ECALECALECALECALTRDTRDTRDTRD
RICHRICHRICHRICHSTSSTSSTSSTS
STSSTSSTSSTS
25-26 March 2004, München25-26 March 2004, München Ivan Kisel, KIP, Uni-HeidelbergIvan Kisel, KIP, Uni-Heidelberg 1414
AlgorithmsAlgorithms
SimpleLocalParallelFast
Hough TransformCellular AutomatonElastic NetKalman Filter
25-26 March 2004, München25-26 March 2004, München Ivan Kisel, KIP, Uni-HeidelbergIvan Kisel, KIP, Uni-Heidelberg 1515
Trigger SimulationTrigger Simulation
L1 FPGA C++, SystemC, VHDL, Synopsys, Quartus
L2 CPU Framework, C++, GEANT
Offline Framework, C++, GEANT
Overall behavior SystemC, Ptolemy
25-26 March 2004, München25-26 March 2004, München Ivan Kisel, KIP, Uni-HeidelbergIvan Kisel, KIP, Uni-Heidelberg 1616
SummarySummary
Modular structure of DAQ Online reconfigurable farm Commercial hardware Modular software „All-in-one“ sub-farm No canonical event building Access to all data at any stage 4D event reconstruction Flexible L1/L2/Offline definition No need to re-process L1 at L2 Offline can run/test L1 and L2 All data on local mass storage Online alignment Online database update