Post on 24-Jan-2016
description
transcript
LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles13 September 2004 1
Performance of the CMS Silicon Tracker Performance of the CMS Silicon Tracker Front-End Driver Front-End Driver
10th Workshop on Electronics for LHC 10th Workshop on Electronics for LHC Experiments and Future ExperimentsExperiments and Future Experiments
R. Bainbridge, E. Corrin, C.Foudas, J. Fulcher, G. Hall, G. Iles, J. Leaver, M. Noy, M. Raymond, O. Zorba
Imperial College
D. Ballard, I. Church, J.A.Coughlan, C.P.Day, E.J.Freeman, W.J.F.Gannon,R.N.J. Halsall, M. Pearson, G. Rogers, J. Salisbury, S. Taghavi, I.R.Tomalin
CCLRC Rutherford Appleton Laboratory
I. ReidBrunel University
Presented by Greg Iles: gm.iles@imperial.ac.uk
LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles13 September 2004 2
Microstrip Tracker readout chainMicrostrip Tracker readout chain
Off Detector (counting room)Off Detector (counting room)
– Optical links transmit equivalent to 1.3 TB/s @ 100kHz trigger rate
– 440 Front End Drivers (FEDs)
On Detector:On Detector:
– 9M silicon strips
– 73k APV25 readout chips
– Analogue readout via 43k optical readout links
APV
MUX
2:1
APV readout chip
128:1
PLL
DCU
APV readout chip
128:1
Front End ModuleDetector
FEDADC
ADC
x12
FP
GA
ADC
ADC
x12
FP
GA
x8
FP
GA
RA
M
Transition card
S-link card
DAQ
FMMThrottle signals
AOH
9U VME back plane
x96
LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles13 September 2004 3
Front End Driver (FED)Front End Driver (FED)
96 optical fibres inputs, each a multiplexed pair of APVs
8 front end blocks each driven by a 12 way optical ribbon cable
Raw input data rate (all 96 fibres)
= 3.4GB/s.
Output rate down slink
= 50MB/s /% occupancy
VME FPGA
Front-End data processing FPGA
Power
S-Link
Back End “System” FPGA
FE Unit
LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles13 September 2004 4
Front End (FE) Unit on FEDFront End (FE) Unit on FED
Opto-RX, 12 way
12 x Buffers
3 x Delay FPGA(ADC clk timing)
6 x Dual 40MHz, 10bit ADCs
Virtex II, 2M gate FPGA performs signal processing
Optical ribbon cable input
Analogue circuitry duplicated on secondary side
Sig
nal
mag
nit
ud
e
Digital header
128 analogue values (one for each microstrip)
MIP
De-multiplexed fibre channel = APV Data FrameDe-multiplexed fibre channel = APV Data FrameTo extract hit need to perform:
- Common mode subtraction
- Pedestal subtraction - Cluster finding - Sync checking
Opto-to-electrical conversion Digitise & sync data Find hit clusters
Time
LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles13 September 2004 5
FED statusFED status
At LECC ’03 in AmsterdamAt LECC ’03 in Amsterdam– Start ‘03: The first two FEDv1 boards were manufactured– June ‘03: After testing showed there were no major faults a further 3 FEDv1 boards were produced
Progress in the last yearProgress in the last year– Sept. ‘03: Further 6 boards had serious problems– Start ‘04: A further batch of 6 boards manufactured and assembled at
different company– Spring ‘04: FEDs distributed to CERN, Pisa, Lyon– June ‘04: Beam test at CERN– Sept. ‘04: FEDv2 should return from manufacture.– End ‘04: Manufacture a further 20 FEDv2 assuming no surprises.
Software and DAQ for the CMS Silicon Tracker Front End Driver Software and DAQ for the CMS Silicon Tracker Front End Driver Poster by Jon FulcherPoster by Jon Fulcher
The Manufacture of the CMS Tracker Front-End DriverThe Manufacture of the CMS Tracker Front-End Driver
Poster by John CoughlanPoster by John Coughlan
LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles13 September 2004 6
Beam Test June ‘04Beam Test June ‘04
CERN X5 AreaCERN X5 Area– 4 FEDs– 252 fibres– 65,000 strips– 2 FEDs slink
– Provided excellent opportunity for system integration
– Large complex system useful for finding system weak points.
Beam
LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles13 September 2004 7
Testing in the labTesting in the lab
FEDFED– Tested with FED
Tester Ensemble
FED Tester EnsembleFED Tester Ensemble– Drives all 96 FED
channels with data similar to that expected in CMS.
– 100kHz Poisson L1A.
– Also provides clock, L1A and throttling from built in Trigger Control System (TCS)
– An ensemble is made up of 4 FED Testers.
Master FED Tester: Provides clock and control to additional 3 FED Testers
FED under test sandwiched between 2 additional FEDs and crate closed to simulate airflow & temperatures in fully populated crate.
VME access via SBS 620 PCI-VME bridge
LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles13 September 2004 8
The FED TesterThe FED Tester
DACs
System FPGA
VME FPGA
AOHs
x-point switches and buffers on back side
Power
Master & slave I/O
Optical outputs
Fibre spools
Data for 3 channels loaded into FPGA.
Converted to analogue form by 3 DACs.
Cross-point switch controls distribution of the 3 unique channels to the 24 channels.
8 three channel TOB type AOHs convert the electrical signal to optical signals.
Temp of AOHs controlled to +/-1ºC
Provides 24 optical channels & Trigger Control System (TCS)Provides 24 optical channels & Trigger Control System (TCS)
LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles13 September 2004 9
Laboratory test set-upLaboratory test set-up
Slink Rx
Generic PCI Card
VME crate PC
Slink PC (PCI-X slots)
Access VME with SBS620 PCI-VME link
LVDS cableFED
Simulate Local Trigger Control System
FT (master)
FT (slave)
FT (slave)
FT (slave)
Slink-Tx
Clock & L1As from FT to FEDThrottle signals from FED to FT
Merg
e 9
6 fi
bre
s in
to 8
rib
bon
s o
f 12
fib
res
J0 Connector
J1 Connector
VME crate
J2 Connector
LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles13 September 2004 10
Slink transition cardSlink transition card
Carrier for slink transmitterCarrier for slink transmitter– Buffers control and data
signals– Buffers throttle signals – FED v1 and V2 compatible
StatusStatus– Returned from
manufacture in August.
– S-link verified– Throttle signals still
under test
DAQ – Slink Transmitter
FED
Slink data and control signals
Throttle signals to Fast Merge Module (FMM)
VM
E B
ackp
lan
e
LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles13 September 2004 11
Slink verificationSlink verification
CheckCheck– Verify data transmitted by the FED via slink is not corrupted.
Set-UpSet-Up– FED sending test patterns (scope mode, sample size 6)– 100 kHz repetitive trigger
• CPU can only verify data transmitted at ~28 Mbytes/sec• Hence FED asserts “BUSY” and trigger rate falls to ~17.1 kHz
ResultsResults– Verified 1 TB of data in 10.8 hrs
• No errors– No errors observed so far, however it would take 146 days to
guarantee no more than 1 error per week in CMS.
However....However....– In rate test with transition card borrowed from
another sub detector (no signal buffering) we did observe occasional errors.
– More tests needed. Add CRC check.
LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles13 September 2004 12
700
600
500
400
300
200
100
0
Slin
k dat
a ra
te (
MB/s
)
6543210
Event size (kB)
Measured rate
Rate if L1As not throttled
Slink max data throughputSlink max data throughput
FED configurationFED configuration– Scope mode, repetitive
triggers at 100kHz.– Varied scope sample length to
vary event size.
Results & ConclusionsResults & Conclusions– Maximum transfer rate =
469MB/s– Observed S-link receiver
exerting back pressure for events > 4.88kB
– PC rather than FED setting upper limit.
– Switched to random triggers at 100kHz
• OK in scope mode & zero suppressed mode
Requirements for CMSRequirements for CMS– 200MB/s Average– 400MB/s Peak
LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles13 September 2004 13
Temperature MonitoringTemperature Monitoring
Fan direction
Air deflector
Simple air deflector in centre of card lowers temperature in hottest region by ~10°C
OptoRx #7
OptoRx #0
Max temp of OptoRx = 70°C
Measurements from on-board sensors and thermocouples
70
60
50
40
30
20
Tem
pera
ture
ºC
Fan speed = 2460 RPM
, No air deflector
, Air defelector
70
60
50
40
30
20
Tem
pera
ture
ºC
6420OptoRx
Fan speed = 3480 RPM
LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles13 September 2004 14
Where are Tracker buffers ?Where are Tracker buffers ?
Buf 1
Buf 12
Front-End FPGA 1
Buf 1
Buf 12
Chan 1
Chan 12
Head Buf
Back-End FPGA
Mux
BE B
uf
Bus 1
Bus 8
80MB/s
200 MB/s Avg
400 MB/s Peak
640 MB/s Unlimited
APV
40 MS/s @ 10bit
Chan 1
Chan 12
Laser Driver
40 MS/s @ 10bit
APV Buffers, ~10 in decon mode
controlled by APVE
FE Buffers, 2kB, ~125 ZS eventsBE Buffers, 2MB, ~1000 ZS events
Front-End FPGA 8
Header Buffer
Quad Data Rate SRAM. Handles 640MB/s in/out simultaneously.
LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles13 September 2004 15
Buffer testsBuffer tests
Check FED buffers can handle high rates & occupanciesCheck FED buffers can handle high rates & occupancies– Want to test “unconstrained” FED
• Don’t want slink back pressure limiting results• Slink throttle disabled.• Data sent to slink oblivion
– Ideally would like all FED buffers to assert “Busy” or “Warn” when becoming full.
• If buffer overflow inevitable -> Detect the event, but ignore data• Set flag to indicate data loss and record the number of these
events
– Not possible yet, although BE buffer can assert throttle signals• Use different approach. Not perfect, but still informative.• No throttling used• Count number of triggers before FED hangs
LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles13 September 2004 16
Buffer test resultsBuffer test results
Conditions:Conditions:Data taken in Zero Suppression mode
Single strip clusters used to create largest event size possible and thus worst case.
APV frame occupancy kept constant. - e.g. 6 strips = 4.7% occupancy - Easier to understand results. - Large buffers -> Valid approx
Performed 5000 "tests“. Each comprised 100k triggers, 100kHz Poisson distributed
Results:Results:FED handles single strip occupancy up to 6.25% (8 single strip clusters)
100
101
102
103
104
105
106
Mea
n lif
etim
e [n
umber
of
trig
gers
]
1612840
Occupancy [%]
Poisson-distributedtriggers (100 kHz mean)
LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles13 September 2004 17
Deadtime Versus OccupancyDeadtime Versus Occupancy
FED Tester EnsembleFED Tester Ensemble– Sending isolated hits– (i.e single strip
clusters)– 2x106 Poisson L1As @
100kHz
FEDFED– Currently transmits
48MB/s of debug information in zero suppressed mode in addition to 32MB/s of status information.
SlinkSlink– Limited to average
rate of 248 MB/s
CMSCMS– Maximum = 3%
occupancy, but large cluster width > 1
60
50
40
30
20
10
0
Dea
dti
me
(Tri
gger
s V
etoe
d %
)
543210
Occupancy based on single strip clusters (%)
3.53.02.52.01.51.00.5
Measured average event size minus debug inf o (kB)
Slink operating at 248MB/ s which is equivalentto 200MB/ s without FED debug information
Worst case because FED uses data f ormat, which is ineffi cient f or single strip clusters. Need more realistic "f ake" data to simulate CMS high ocupancy regions.
LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles13 September 2004 18
Cluster width extrapolationCluster width extrapolation
Cluster algorithmCluster algorithm– Following is simplification,
however:• 1 byte pos• 1 byte width• 1 byte for every hit strip
Calculate average event sizeCalculate average event size– Make some simplifications
• Ignore data padding for byte aligned data
• Ignore 2 dead channels
Beam test June 2004Beam test June 2004– Average cluster width = 1.29 strips– Average cluster event size = 3.29 bytes
100
80
60
40
20
0
Dea
dti
me
(Tri
gger
s V
etoe
d %
)
43210
Measured average event size minus debug inf o (kB)
Measured deadtime
Average event size based on 3% occupancy
Measured: single strip clusters Theory: single strip clusters Theory: beam test clusters Theory: fi xed cluster width of 2
LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles13 September 2004 19
SummarySummary
FED v1FED v1– Commissioning tests at RAL, Imperial & CERN continuing well.– Baseline firmware and software operational.
FED v2 and S-link Transition cardFED v2 and S-link Transition card– Both back from manufacture in August ’04– Only preliminary results so far, but everything seems OK
FutureFuture– System integration
• Calibration of Tracker, Databases, Large system issues (e.g. error handling)– Production issues
• Tender, Industrial testing– Continued testing
• Beam test in October, Continued testing with the Fed Tester.– Commissioning
• Start to plan
LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles13 September 2004 20
More information...More information...
Software and DAQ for the CMS Silicon Tracker Front End Driver Software and DAQ for the CMS Silicon Tracker Front End Driver Poster by Jon FulcherPoster by Jon Fulcher
The Manufacture of the CMS Tracker Front-End DriverThe Manufacture of the CMS Tracker Front-End Driver
Poster by John CoughlanPoster by John Coughlan