Date post: | 18-Jan-2016 |
Category: |
Documents |
Upload: | madison-bruce |
View: | 218 times |
Download: | 0 times |
Sep. 17, 2002Sep. 17, 2002 BESIII Review MeetingBESIII Review Meeting
BESIII DAQ SystemBESIII DAQ System
BESIII Review MeetingBESIII Review Meeting
IHEP · Beijing · China Sep. 16 - 17, 2002IHEP · Beijing · China Sep. 16 - 17, 2002
Sep. 17, 2002 BESIII Review Meeting
BESIII DAQ Scheme
. . .
Branch 1
PowerPC VME Modul e
. . .
Tape
Branch M
.. .
VME Crate VME Crate
Tri gger System
.. .
VME Crate. . .
100MEthernet
1000MEthernet
Tri ggerSi gnal
Fi l e Server
Farm Node 1 Farm Node 2 Farm Node N
1000M Swi tch
Readout PC
100M Swi tch
Readout PC
100M Swi tch
Readout PC
100M Swi tch
Farm Supervi sor
Run Control EventDi spl ay
Hi stogramDi spl ay
Sl ow Control
100M Swi tch
Sep. 17, 2002Sep. 17, 2002 BESIII Review MeetingBESIII Review Meeting
Readout Crate System Controller – MVME2431-1Readout Crate System Controller – MVME2431-1
PowerPC 750™ 32-bit PowerPC 750™ 32-bit microprocessormicroprocessor
32MB of on-board ECC SDRAM32MB of on-board ECC SDRAM Up to 1MB capacity for on-board Up to 1MB capacity for on-board
firmware firmware 8MB on-board Flash memory 8MB on-board Flash memory One asynchronous serial debug One asynchronous serial debug
portport 10/100Mbps Ethernet interface10/100Mbps Ethernet interface 4-level requester, 7-level 4-level requester, 7-level
interrupter, and 7-level interrupt interrupter, and 7-level interrupt handler for VMEbushandler for VMEbus
Sep. 17, 2002Sep. 17, 2002 BESIII Review MeetingBESIII Review Meeting
3 Levels of Event Builder3 Levels of Event Builder
... ... ...
... ...
Slice
Fragment
Sub-event
Event
Event Bui l der
FEEs
ReadoutCrates
Readout PCs(Branch)
OnlineFarm
Hardware
Level-1 trigger
1 GbSwitch
100 MbSwitch
Sep. 17, 2002Sep. 17, 2002 BESIII Review MeetingBESIII Review Meeting
BESIII DAQ TasksBESIII DAQ Tasks Event Data Readout from FEEEvent Data Readout from FEE Event BuildingEvent Building Event FilteringEvent Filtering Event Recording to Persistent MediaEvent Recording to Persistent Media Run Control SystemRun Control System Monitoring Monitoring (event, histogram display ...)(event, histogram display ...) Message ReportingMessage Reporting
Sep. 17, 2002Sep. 17, 2002 BESIII Review MeetingBESIII Review Meeting
Data Rate EstimationData Rate Estimation Level-1 Trigger Rate: 4KHz Level-1 Trigger Rate: 4KHz (2KHz Good (2KHz Good
Events)Events)
Event Size: 12KBytesEvent Size: 12KBytes
Data Rate to be Processed by DAQ: Data Rate to be Processed by DAQ: 48MByte/sec48MByte/sec
To be Recorded on Disk Array/Tape: To be Recorded on Disk Array/Tape: 36MByte/sec36MByte/sec
Sep. 17, 2002Sep. 17, 2002 BESIII Review MeetingBESIII Review Meeting
TechnologiesTechnologiesMulti-level BufferingMulti-level BufferingSwitch NetworkSwitch NetworkParallel ComputingParallel ComputingEasily Upgrade and PortEasily Upgrade and PortEasily ScaleEasily ScaleReliable / Stable / Error RecoveryReliable / Stable / Error Recovery
Sep. 17, 2002Sep. 17, 2002 BESIII Review MeetingBESIII Review Meeting
Online Data FlowOnline Data FlowFarm Node 1
Readout PC
111
2
3 3 3
4 5
6
666
Farm Supervi sor1000M Swi tch
100M Swi tch
Run Control Event Di spl ay
Hi stogram Di spl ay
Fi l e Server
Tape
6
3
Network Connecti on
Control Command
Event Data
Farm Node 2 Farm Node N
A B C D E F G HSELECTED
ON-LINE
A B C D E F G HSELECTED
ON-LINE
Sep. 17, 2002Sep. 17, 2002 BESIII Review MeetingBESIII Review Meeting
PowerPC/VxWorks Based Test EnvironmentPowerPC/VxWorks Based Test Environment
Sep. 17, 2002Sep. 17, 2002 BESIII Review MeetingBESIII Review Meeting
Test Results – Programmed I/OTest Results – Programmed I/O
System Controller: MVME2431-1 (350 MHz System Controller: MVME2431-1 (350 MHz PowerPCPowerPC 750) 750)
Operating System: VxWorks 5.4Operating System: VxWorks 5.4
Writing from CPU DRAM into VMEbusWriting from CPU DRAM into VMEbus
32-bit transfers: 400 ns/32-bit write,32-bit transfers: 400 ns/32-bit write, 10MB/s 10MB/s
16-bit transfers: 400 ns/16-bit write, 16-bit transfers: 400 ns/16-bit write, 5MB/s5MB/s
Reading from VMEbus into CPU DRAMReading from VMEbus into CPU DRAM
32-bit transfers: 1,100 ns/32-bit read,32-bit transfers: 1,100 ns/32-bit read, 3.6MB/s 3.6MB/s
16-bit transfers: 1,100 ns/16-bit read,16-bit transfers: 1,100 ns/16-bit read, 1.8MB/s 1.8MB/s
Sep. 17, 2002Sep. 17, 2002 BESIII Review MeetingBESIII Review Meeting
Note:Note:
The above VME access speeds were The above VME access speeds were Programmed I/O speeds.Programmed I/O speeds.
The program was written in C language and The program was written in C language and
run in the context of the VxWorks shell.run in the context of the VxWorks shell.
The tests were conducted on an idle system The tests were conducted on an idle system
with no other competing processes nor any with no other competing processes nor any
concurrent activity on the VMEbus. concurrent activity on the VMEbus.
Sep. 17, 2002Sep. 17, 2002 BESIII Review MeetingBESIII Review Meeting
Question: Question:
Can we use Can we use Programmed ReadProgrammed Read to read to read out data from ADCs/TDCs modules?out data from ADCs/TDCs modules?
Sep. 17, 2002Sep. 17, 2002 BESIII Review MeetingBESIII Review Meeting
Estimation of One VME Readout Crate (1)Estimation of One VME Readout Crate (1)
Basic Assumptions:Basic Assumptions: 1 crate = 1 crate = 1616 modules modules 1 module = 1 module = 3232 channels channels Trigger rate: Trigger rate: 4000 Hz4000 Hz When one L1 trigger comes, When one L1 trigger comes, 10%10% channels have channels have
signals.signals. Each channel has its own Buffer containing Each channel has its own Buffer containing 16-bit 16-bit
datadata to be read. to be read. Each module has a Hit Map (Each module has a Hit Map (32-bit32-bit) ) Data readout is in Data readout is in A24/D16A24/D16 mode mode and 1.1 and 1.1
microsec/16bit readmicrosec/16bit read
Sep. 17, 2002Sep. 17, 2002 BESIII Review MeetingBESIII Review Meeting
Estimation of One VME Readout Crate (1) Estimation of One VME Readout Crate (1) (con’t)(con’t)
For each module:For each module:Data Ready Flag:Data Ready Flag: 1 word1 word
Trigger Number:Trigger Number: 1 word1 word
Hit Map:Hit Map: 2 word 2 word
Data:Data: 32 × 10% = 3.2 words32 × 10% = 3.2 words
For each crate:For each crate: 16 × ( 1 + 1 + 2 + 3.2 ) = 115.2 words16 × ( 1 + 1 + 2 + 3.2 ) = 115.2 words
When trigger rate is 4000 Hz, readout time will be:When trigger rate is 4000 Hz, readout time will be:
4000 × 115.2 bytes 4000 × 115.2 bytes ×× 1.1 microsec = 1.1 microsec = 0.51 sec0.51 sec
Sep. 17, 2002Sep. 17, 2002 BESIII Review MeetingBESIII Review Meeting
Estimation of One VME Readout Crate (2)Estimation of One VME Readout Crate (2)
Basic Assumptions:Basic Assumptions: 1 crate = 1 crate = 1616 modules modules 1 module = 1 module = 3232 channelschannels Trigger rate: Trigger rate: 4000 Hz4000 Hz When one L1 trigger comes, When one L1 trigger comes, 10%10% channels have channels have
signals.signals. Each module has a Each module has a Global BufferGlobal Buffer containing containing 16-16-
bit channel numberbit channel number and and 16-bit data16-bit data to be read. to be read. Data readout is in Data readout is in A24/D32A24/D32 mode mode and and 1.11.1
microsec/32bit readmicrosec/32bit read
Sep. 17, 2002Sep. 17, 2002 BESIII Review MeetingBESIII Review Meeting
Global Buffer Data FormatGlobal Buffer Data Format31 031 0
……
Event DataEvent DataChannel NumberChannel Number
…………
…………
TailTail
……
Event DataEvent DataChannel NumberChannel Number
Trigger NumberTrigger Number
TailTail
Event DataEvent DataChannel NumberChannel Number
Event DataEvent DataChannel NumberChannel Number
Trigger NumberTrigger Number
Sep. 17, 2002Sep. 17, 2002 BESIII Review MeetingBESIII Review Meeting
Estimation of One VME Readout Crate (2)Estimation of One VME Readout Crate (2)(con’t)(con’t)
For each module:For each module:
Data Ready Flag:Data Ready Flag: 1 DWord1 DWord
Header:Header: 1 DWord 1 DWord
Data:Data: 32 × 10% = 3.2 32 × 10% = 3.2 DWordsDWords
Tail:Tail: 1 DWord1 DWord
For each crate:For each crate:
16 × ( 1 + 1 + 3.2 + 1 ) = 99.2 DWords16 × ( 1 + 1 + 3.2 + 1 ) = 99.2 DWords
When trigger rate is 4000 Hz, readout time will be:When trigger rate is 4000 Hz, readout time will be:
4000 × 99.2 bytes 4000 × 99.2 bytes ×× 1.1 microsec = 1.1 microsec = 0.44 sec0.44 sec
Sep. 17, 2002Sep. 17, 2002 BESIII Review MeetingBESIII Review Meeting
So, the answer is:So, the answer is:
The system controller does not have enough The system controller does not have enough time to read out data from ADCs/TDCs by time to read out data from ADCs/TDCs by using Programmed Read.using Programmed Read.
The way to get faster reads is to use the The way to get faster reads is to use the DMA engine resident on the Universe VME DMA engine resident on the Universe VME <=> PCI bridge chip. It is designed to copy <=> PCI bridge chip. It is designed to copy larger chunks of data from one bus to the larger chunks of data from one bus to the other. other.
Sep. 17, 2002Sep. 17, 2002 BESIII Review MeetingBESIII Review Meeting
MVME2431 DMA ControllerMVME2431 DMA Controller The Universe II has a DMA controller for high performance The Universe II has a DMA controller for high performance
data transfer between the PCI bus and VMEbus.data transfer between the PCI bus and VMEbus. Transfer sizeTransfer size
Any Number of bytes from Any Number of bytes from 1 byte to 16 Mbytes1 byte to 16 Mbytes.. Transfer Data WidthTransfer Data Width
D08 (EO)D08 (EO) D16D16 D16BLTD16BLT D32D32 D64D64 D32BLTD32BLT D64BLT (MBLT)D64BLT (MBLT)
Sep. 17, 2002Sep. 17, 2002 BESIII Review MeetingBESIII Review Meeting
Readout Mode: Global Buffer + DMAReadout Mode: Global Buffer + DMA
Block size must be counted Block size must be counted somewhere on board. somewhere on board.
To avoid too many DMA To avoid too many DMA operations (DMA Interrupts), operations (DMA Interrupts), it’s better for each ADC/TDC it’s better for each ADC/TDC board to board to buffer data of buffer data of multiple triggersmultiple triggers (e.g. 64, 128 (e.g. 64, 128 triggers), and then triggers), and then interruptinterrupt the system controller to start a the system controller to start a DMA operation.DMA operation.
Sep. 17, 2002Sep. 17, 2002 BESIII Review MeetingBESIII Review Meeting
DMA TransferDMA Transfer
Some typical values for DMA transfers:Some typical values for DMA transfers:
D32 Read:D32 Read: 20 -- 23 MB/s20 -- 23 MB/s
D32 Write:D32 Write: 22 -- 25 MB/s22 -- 25 MB/s
D64 Read:D64 Read: 40 -- 45 MB/s40 -- 45 MB/s
D64 Write:D64 Write: 45 -- 50 MB/s45 -- 50 MB/s
Note:Note: This is the raw speed of the hardware This is the raw speed of the hardware without any software overhead.without any software overhead.
Sep. 17, 2002Sep. 17, 2002 BESIII Review MeetingBESIII Review Meeting
Main Tasks of ReadoutMain Tasks of Readout
DMA transferDMA transfer CalculationCalculation Data unpacking & packingData unpacking & packing Network transmission.Network transmission.
Sep. 17, 2002Sep. 17, 2002 BESIII Review MeetingBESIII Review Meeting
R & D PrototypeR & D Prototype
Online Farm
Node 1 Node 2
ReadoutPCs
Farm Supervisor
100Mbps Switch100Mbps Switch
Single EventDisplay
File Server
A B C D E F G HSELECTED
ON-LINE
A B C D E F G HSELECTED
ON-LINE
Node 1 Node 2
Sep. 17, 2002Sep. 17, 2002 BESIII Review MeetingBESIII Review Meeting
File ServerFile Server
DELL PowerEdge 2500 Dual Intel Pentium III DELL PowerEdge 2500 Dual Intel Pentium III 1.4 GHz Processors with 512K Cache1.4 GHz Processors with 512K Cache
Integrated 10/100 Ethernet AdapterIntegrated 10/100 Ethernet Adapter 1 GBytes SDRAM1 GBytes SDRAM SCSI RAID ControllerSCSI RAID Controller Six 18GB SCSI Hard Drive, 10K RPMSix 18GB SCSI Hard Drive, 10K RPM Intel Pro 1000 Fiber Network Adapter Intel Pro 1000 Fiber Network Adapter Red Hat Linux 7.3 Kernel 2.4.18-3 smpRed Hat Linux 7.3 Kernel 2.4.18-3 smp
Sep. 17, 2002Sep. 17, 2002 BESIII Review MeetingBESIII Review Meeting
Online Farm & Readout PC NodesOnline Farm & Readout PC Nodes
DELL GX260 Intel Pentium 4 1.8 GHz ProcesDELL GX260 Intel Pentium 4 1.8 GHz Processor with 256K Cachesor with 256K Cache
Integrated 10/100/1000 Ethernet AdapterIntegrated 10/100/1000 Ethernet Adapter 512 MBytes SDRAM512 MBytes SDRAM 40GB EIDE 7200 RPM Hard Drive40GB EIDE 7200 RPM Hard Drive Red Hat Linux 7.3 Kernel 2.4.18-3Red Hat Linux 7.3 Kernel 2.4.18-3
Sep. 17, 2002Sep. 17, 2002 BESIII Review MeetingBESIII Review Meeting
Ethernet SwitchEthernet Switch
Catalyst 2948G-L3Catalyst 2948G-L3 48 Ports of 10/100-Mbps Fast Ethernet48 Ports of 10/100-Mbps Fast Ethernet 2 Ports of 1-Gbps Gigabit Ethernet2 Ports of 1-Gbps Gigabit Ethernet
Sep. 17, 2002Sep. 17, 2002 BESIII Review MeetingBESIII Review Meeting
Network Performance Network Performance (Preliminary)(Preliminary)
0
20
40
60
80
100
0 32 64 96 128
Bl ock si ze (KBytes)
Thro
ughp
ut (
Mbps
)
Block sizeBlock size ThroughputThroughput Block sizeBlock size ThroughputThroughput Block sizeBlock size ThroughputThroughput
(Bytes)(Bytes) (Mbps)(Mbps) (Bytes)(Bytes) (Mbps)(Mbps) (Bytes)(Bytes) (Mbps)(Mbps)
88 0.470.47 10241024 26.1526.15 131072131072 87.6987.69
1616 0.930.93 20482048 36.5136.51 262144262144 88.7788.77
3232 1.821.82 40964096 52.5852.58 524288524288 89.2689.26
6464 3.543.54 81928192 65.6065.60 10485761048576 89.5289.52
128128 6.546.54 1638416384 75.7675.76 20971522097152 89.6689.66
256256 11.8111.81 3276832768 81.9181.91 41943044194304 89.7289.72
512512 18.6118.61 6553665536 85.9685.96 83886088388608 89.7689.76
Sep. 17, 2002Sep. 17, 2002 BESIII Review MeetingBESIII Review Meeting
More Studies on PrototypeMore Studies on Prototype Performance StudiesPerformance Studies
Network performance in parallel modeNetwork performance in parallel mode Raid disk performanceRaid disk performance
write/read speed vs. config (raid0/raid5)write/read speed vs. config (raid0/raid5) Data flow control softwareData flow control software Online filter softwareOnline filter software Distributed histogram processingDistributed histogram processing Single event displaySingle event display
Sep. 17, 2002Sep. 17, 2002 BESIII Review MeetingBESIII Review Meeting
Thanks!Thanks!