Date post: | 14-Jan-2016 |
Category: |
Documents |
Upload: | kenneth-norman |
View: | 212 times |
Download: | 0 times |
J. Coughlan et al. 1st October 2003LECC 2003 Amsterdam
The CMS Tracker Front-End Driver
9th Workshop on Electronics for LHC Experiments Amsterdam
J.A.Coughlan, S.A. Baird, I. Church, C.P.Day, E.J.Freeman, W.J.F.Gannon,R.N.J. Halsall, M. Pearson, G. Rogers, J. Salisbury, S.Taghavirad, I.R.Tomalin
CCLRC Rutherford Appleton Laboratory
E. Corrin, C.Foudas, J. Fulcher, G. Hall, G. Iles, M. Noy, O. ZorbaImperial College
I. ReidBrunel University
Presented by John [email protected]
J. Coughlan et al. 1st October 2003LECC 2003 Amsterdam
CMS Tracker FED FEDv1
This TalkOverview of FEDv1 ArchitectureBoard Status &Test Results@ RAL & Imperial College
Primary Function
Data reduction and event building.To extract hit strip information from CMS Silicon Tracker FEnds
J. Coughlan et al. 1st October 2003LECC 2003 Amsterdam
CMS Tracker FEDSilicon Strip Tracker Readout Overview
~ 9 million Silicon Strip channels
ON Detector: 73K APV25 pipeline chips
@ L1 Trigger: MUX APV Frame output
Analogue Data readout via Optical links
(APV Frame: Header + Strip Data)
OFF Detector: Front-End Drivers (FED)
Digitize / Zero Suppress / DAQ readout
440 x 9U VME64x boards
96 ADC channel boards
DAQ
CountingRoom
On Detector
FPGA
25
VME 9U FEDs
HybridFront-End HybridSilicon Strips
J. Coughlan et al. 1st October 2003LECC 2003 Amsterdam
CMS Tracker FED FED Layout
VME-FPGA
TTCrx
BE-FPGAEvent Builder
Buffers
FPGAConfiguration
PowerDC-DC
DAQInterface
12
12
12
12
12
12
12
12
Front-End Modules x 8Double-sided board
CERNOpto-
Rx Analogue/Digital
96 TrackerOpto Fibres
VMEInterface
XilinxVirtex-IIFPGA
Modularity
9U VME64x Form Factor
Modularity matches Opto Links
8 x Front-End “units”
OptoRx/Digitisation/Cluster Finding
Back-End module / Event Builder
VME module / Configuration
Power module
Other Interfaces:
TTC : Clk / L1 / BX
DAQ : Fast Readout Link
TCS : Busy & Throttle
VME : Control & Monitoring
JTAG : Test & Configuration
FE-FPGAClusterFinder
TTC
TCS
TempMonitor
JTAG
TCS : Trigger Control System
9U VME64x
J. Coughlan et al. 1st October 2003LECC 2003 Amsterdam
CMS Tracker FED First Module January - March 2003
“Primary” Side
• FEDs under test:
Ser001: Imperial 2 ORx installed Marchdetailed characterisation of FED with optical inputs
Ser002: RALFirmware development / Electrical / digital tests
Ser 003-005 assembled JuneSer 005 with 8 ORx’s
Ser 006-011PCBs due SeptemberTo be fully equipped with ORx’s (1 month)
OptoRx
CFlash
VME64x9U board
34 xFPGAs
Analogue
TTC
FE Unit
Power
Memories
96 channels
JTAG
J. Coughlan et al. 1st October 2003LECC 2003 Amsterdam
CMS Tracker FED FPGAs & Firmware
Delay x 24 FE x 8
BE x 1
VME x 1
Delay FPGA: ADC Coarse and Fine Clock Skewing.
FE FPGA: Scope and Frame Finding modes.
BE FPGA: Event building, buffering and formatting.
VME FPGA: Controls and Slow Readout path.
3434 FPGAsFPGAs on board
NB Final Firmware was only implementedafter board manufacture. Now ready.
Test Firmware used for early results
40k -> 2M gates40k -> 2M gates
Xilinx Virtex-IIXilinx Virtex-II
FPGA ConfigurationFPGA ConfigurationSystem ACE Compact FlashSystem ACE Compact Flashthru JTAG chainthru JTAG chain
J. Coughlan et al. 1st October 2003LECC 2003 Amsterdam
CMS Tracker FED First Fully Assembled Board June 2003
Primary Side Secondary Side
9U VME64x
PCB (2mm) 14 layers (incl 6 power & ground)
96 ADC channels : AD9218 Dual package 10 bit @ 40 MHz
~ 6 K components ; ~ 25 K tracks
1/2 Analogue circuitry on Secondary Side
High density at Front-End Units
FE Unit
BGAs 676 pins @ 1 mm pitch
JTAG Boundary Scan
J. Coughlan et al. 1st October 2003LECC 2003 Amsterdam
CMS Tracker FED Zoom in on FE Unit
Front-End Unit = 12 channels
DualADCs
OpAmps
DelayFPGAs
Duplicated onSecondary Side
“OptoRx”
“Primary” Side
TestConnector
Resistor Packs
“OptoRx” modules CERN projectCommercial Package with PIN Diode + Custom Analogue ASICCustom Analogue ASIC
TrimDAC
J. Coughlan et al. 1st October 2003LECC 2003 Amsterdam
CMS Tracker FED Cross-point switch test card
uses AD8116 16x16analogue X-point switch
Electrical testing:ie Pre OptoRx modules
1 card per FE Unit3 inputs to any of 12 FED channels
Chain 8 cards to cover whole FED
J. Coughlan et al. 1st October 2003LECC 2003 Amsterdam
CMS Tracker FED Early Electrical Tests in FEDv1
0
200
400
600
800
1000
1200
1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89
Series1
Series2
Series3
Series4
Series5
Series6
Series7
Series8
Series9
Series10
Series11
Series12
Chip-Scope Pro Logic Analyser capture10 bit-raw data on 12 channels in FE FPGA
sine input (1 MHz) via Cross-Point Switch test card to 12 channels
first 100 (of 4k) samples @ 40 MHz
ADC count
NB Chip-Scope also invaluable as Firmware Debugger!
Testing analogue circuits, connections etc
J. Coughlan et al. 1st October 2003LECC 2003 Amsterdam
CMS Tracker FED First prototype 6U VME Opto-Tester
VME interface logic
Analogue section
Sequence control logic
Sequence storage
DACAmplification+cmAnalogue opto-tx
Acknowledgements for following results to Acknowledgements for following results to Matthew NoyMatthew Noy , Imperial College , Imperial College
Drives up to 3 Fibres
J. Coughlan et al. 1st October 2003LECC 2003 Amsterdam
CMS Tracker FED Early Analogue Characterisation effects
RMS noise of ~ 1.1 counts ≈ 350e- (including ORx)
ADC? effect under investigation
Ch. 0 23with ORx
Ch. 24 95without ORx
Important that FED meets analogue specifications toodetailed characterisation only possible with complete modules and sophisticated tester (built - also needs evaluation and careful control)
All preliminary results are positive details under study … e.g. ….
Timing scan of 25 nsec pulse
J. Coughlan et al. 1st October 2003LECC 2003 Amsterdam
CMS Tracker FED Other Test Results
Require fine & coarse clock skewingindependently on all 96 FED channels
32 fine steps of ~ 800 psec
Implemented by Virtex-IIDigital Clock Managers
Independent TrimDAC offsets at ADC inputson all 96 FED channels
ADC Linearity plot
J. Coughlan et al. 1st October 2003LECC 2003 Amsterdam
CMS Tracker FED FED Large Scale Opto-Tester
• Relatively sophisticated tester needed to evaluate FEDs fullyneed to compare analogue input with digital outputs
large volume of well defined data must be scrutinised
analogue performance should not degrade systemnoise - characterise complete FED, not just single channels
input data should be very stable - challenging given laser T sensitivity
• Tester system comprises 4 modules, each driving 24 optical channels.
Each module has 3 unique analogue channels, each driven by 12 bit, 40MHz DACs.
Cross-point switches allow the 24 outputs to select any of these 3 channels or an external one.
The FED Tester can act as a Trigger Control System if needed (e.g provide Clk, L1As, BC0, etc)
System control is achieved with 2 Virtex II, 1M gate FPGAs
J. Coughlan et al. 1st October 2003LECC 2003 Amsterdam
CMS Tracker FED FED Large Scale Opto-Tester
DACsFibre reels
8-way MU optical connectors
Clks provided via QPLLs
Analogue Optical Hybrids
Cross-point switches
VME interface
System control FPGA
Awaiting AOHs to complete first 24 channel module(G. Iles, J. Leaver + C. Foudas, O. Zorba)Refer to talk at these proceedings
J. Coughlan et al. 1st October 2003LECC 2003 Amsterdam
CMS Tracker FED First FED Readout of 9U Opto-Tester Data
12
10
8
6
4
2
0
Pai
r of
AP
Vs
350340330320310300290
Clock Count
Now using Final FirmwareFinal Firmwarein all FPGAsFully formatted DAQ eventsDAQ eventsvia VME readout
12th FED channel TrimDAC set to full scale
2nd FED channel disabledChannels 1,3 & 4 delayed with Coarse Skew
Tester input on 12 fibres1 x FE Unit
Scope Mode capture(Frame Finding under test)
APV25 simulated Frames
J. Coughlan et al. 1st October 2003LECC 2003 Amsterdam
CMS Tracker FED Schedule
I) FEDs for Large Scale Assembly (LSA) tests: restricted FED functionality
“October ‘03 (1) / end ‘03 (2) / beg ‘04 (2) / mid ‘04 (6)”
II) FED Pre-Production Manufacture: Q1-2/04
full FED functionality
assumes new design iteration FEDv2 pcbs
Design TestProduction& InstallationPre-Pro
FEDv3 (500)FEDv2 (20)FEDv1 (20)
J. Coughlan et al. 1st October 2003LECC 2003 Amsterdam
CMS Tracker FED Summary
• Testing is progressing well…but much more to do!
No “show stoppers” for Silicon Large Scale Assembly tests - functions operating. Analogue effects still need to be understood.
Expect to deliver first FED to CERN with necessary firmwarefirmware around end of October
Final Firmware with full DAQ Formatted Events readout via VME ready.
• Issues
Major software/firmware effort - but appears to be converging well
Availability of important components
OptoRx - FED is following schedule closely - no shortage
VME crates, controllers, Analogue Optohybrids (late)
FPGAs and other parts in hand for ~20 FEDs (including those assembled)
• Final procurement for Mass Production of 500 FEDs
Yield, assembly quality - and consequent testing - will be main concern
J. Coughlan et al. 1st October 2003LECC 2003 Amsterdam
CMS Tracker FED
J. Coughlan et al. 1st October 2003LECC 2003 Amsterdam
CMS Tracker FED
Static Noise Measurements:2/8 Analogue Opto-Receivers placedProtection caps inserted absence of input signals
D.C. level set by TrimDAC and Opto-rx (if present) search over parameter space
Random triggering from VME
Opto-RX (HXR9003)
TrimDAC
Fully differential op-amp ADC
To readout
J. Coughlan et al. 1st October 2003LECC 2003 Amsterdam
DAQFront-endReadout
LinkFRL
CMS Tracker FED FED-DAQ Interface
FRL DAQ links use S-LINK64 standardImplementation: Channel Link800 MBytes/sec max
Average DAQ rate 200 MBytes/sec
VME-FPGA
TTCrx
Back-End FPGA“Event Builder”
Buffers
FPGAConfiguration
Hot SwapPSU
DC-DC
DAQLink
12
12
12
12
12
12
12
12
CERNOpto-
Rx ADC/Digital
96 TrackerOpto Fibres
Front-End FPGA“Cluster Finder”
DAQ MezzanineCard
Transition Card
TCSEXT CLOCK
S-LINK64
VMEInterface
9U VME64x Card
DAQ
TTC
Temp Monitor
JTAG
J. Coughlan et al. 1st October 2003LECC 2003 Amsterdam
FEDFirst fully assembled board.
• Proceed cautiously
~2 months electrical tests
OptoRx is new component
available March 2003
Limited supply & costly
• June 2003First complete board
• no difficulties to datebut not yet equipped to drive all 96 channels simultaneously
Primary side...
NB Secondary side carries 48 analogue channels...
J. Coughlan et al. 1st October 2003LECC 2003 Amsterdam
FED performance
• Important that FED meets analogue specifications toodetailed characterisation only possible with complete modules and sophisticated
tester (built - also needs evaluation and careful control)
• All preliminary results are positive details under study … e.g. ….
Ch. 0 23with ORx
Ch. 24 95without ORx
RMS noise of ~ 1.1 counts ≈ 350e- (including ORx)
Clock phase adjustment
(M. Noy)
J. Coughlan et al. 1st October 2003LECC 2003 Amsterdam
FEDLarge Scale Assembly Test Requirements
2003“read out Virgin Raw Data formatted as DAQ events via VME in response to TTC trigger and clock.”
Need 96 OptoRx chans. Trigger & Readout rates are not critical.
Functionality
Does require:
•Scope Mode and Software Triggers for set up.
•Controls from VME for run mode, clock source, clock skew, OptoRx offsets (with readback.)
•VME Event buffer with standard DAQ events. Counters for triggers & errors.
•System ACE loading, Clock/Trig/Resets on TTC Chan A, Hardware throttle output.
•FED delivered as a Package including Software Library to drive the Firmware.
Does not require:
S-LINK readout, Clustering mode, Spy Channel, TTC chan B, TCS (but maybe simple throttle), DAC control, pedestal/threshold data, System ACE interface, VME64x config EPROM…, VME Interrupts, Temp chip control…
J. Coughlan et al. 1st October 2003LECC 2003 Amsterdam
CMS Tracker FED FED Layout
VME-FPGA
TTCrx
BE-FPGAEvent Builder
Buffers
FPGAConfiguration
PowerDC-DC
DAQInterface
12
12
12
12
12
12
12
12
Front-End Modules x 8Double-sided board
CERNOpto-
Rx Analogue/Digital
96 TrackerOpto Fibres
VMEInterface
XilinxVirtex-IIFPGA
Digital Processing
Flexible Digital Logic:
Xilinx Virtex-II FPGAs 40K->3M gates*
*some in pin compatible packages
Features:
Dual Ported Block Rams
Digital Clock Managers DCM
Double Data Rate I/O DDR
Digitally Controlled Impedance I/O
Various I/O signal standards
Debugging: Logic Analyser cores
FPGAs programmed in
VHDL & VERILOG
FE-FPGAClusterFinder
TTC
TCS
TempMonitor
JTAG
TCS : Trigger Control System
9U VME64x
J. Coughlan et al. 1st October 2003LECC 2003 Amsterdam
CMS Tracker FED Firmware Status
Clocks
Data
SerialControls
VMELINK
VMEBus
VMESystemSystemACEACE
SystemACE
Clocks
EPROMEPROM
EPROM
TTCrxQDR Write QDR Read
SerialComms
Headers
TTCchanA
VME LinkRegs
S-LINKS-LINK S-LINK
Clocks
Data
SerialComms
ScopeMode
HeaderMode
FIFOs
Input
Regs
SerialControls
ScopeMode
Frame-FindngMode
Output
Input
Regs
Opto Rx DACOpto Rx DAC
DELAY FPGA x 3 x 8
FE FPGA x 8
BE FPGA
VME FPGA
ADCADC
Under Simulation
Under Test on FED
Only for FEDv2Only for FEDv2 Controls
Data Readout
Control
Throttle TCSInput
Cluster FindingCluster FindingModeMode
EdEd
SaeedSaeed
IvanIvan
EdEd
Ed, JohnEd, John
Saeed, IvanSaeed, Ivan
Chan BChan B
I2C
Temp
“Working” on FED
External Devices
TempTemp
14th July 2003
To be Implemented
QDR QDR
J. Coughlan et al. 1st October 2003LECC 2003 Amsterdam
CMS Silicon Strip Tracker FED Front-End module
12 FibreRibbon
PD
Arr
ay
111
2
1
23
4
35
6
2
47
8
59
10
3
611
12
5*
CLK
DCM
LVDS CLK40 from TTC
CERN Opto Rx
6
CLOCK
CONTROL
DATA
DATA OUT@ 160 MHz
3 N
FullPartially Full
RESET
ClusterFindingFPGA
DelayFPGA
DelayFPGA
DelayFPGA
Dual ADC10-bits40 MHz
OpAmp
XC2V1500XC2V40AD9218EL2140
CLK
CLK
10ASIC
4*
DataControl
Digital Processing
12x trim DAC
TempSensorLM82
* Double Data Rate I/O
Each individual ADC clock skewis adjustable in steps ~ 1nsec
J. Coughlan et al. 1st October 2003LECC 2003 Amsterdam
CMS Silicon Strip Tracker FED Front-End FPGA Logic
ADC 1 10 10
sy
nc
11
trig2
Pe
d s
ub
11
trig3
Re
-ord
er
cm
su
b
8
Hit
fin
din
g
s-datas-addr8
16
hit
Pa
ck
eti
se
r
4
averages 8header control
DP
M 16
No hits
Se
qu
en
ce
r-m
ux
8 8a
d
a
d
ADC 1210 10
trig1
sy
nc
11
trig2
Pe
d s
ub
11
trig3
Re
-ord
er
cm
su
b
8
Hit
fin
din
g
s-datas-addr8
16
256 cycles 256 cycles
hit DP
M 16
No hits
Se
qu
en
ce
r-m
ux
8 8a
d
a
d
status
averages8header status
nx256x16
trig4
Synch inSynch out
Sy
nc
h
emulator in
mu
x
Serial I/O
Se
ria
l In
t
B’Scan
Lo
ca
lIO
Config
Cluster Finding FPGA VERILOG Firmware
Full flags
data
Global reset
Co
ntr
ol
Sub resets
10
10
Ph
as
eR
eg
iste
rsP
ha
se
Re
gis
ters
2 x 256 cycles 256 cycles nx256x16
trig1Synch error
4x
Temp Sensor
Delay Line
Opto Rx
Clock 40 MHzD
LL
1x2x4x
per adc channel phase compensation required to bring
data into step
+ Raw Data mode, Scope mode, Test modes...
160 MHz
J. Coughlan et al. 1st October 2003LECC 2003 Amsterdam
CMS Tracker FED FED Data Rates
VME-FPGA
TTCrx
BE-FPGAEvent Builder
Buffers
FPGAConfiguration
PowerDC-DC
DAQInterface
12
12
12
12
12
12
12
12
Front-End Modules x 8Double-sided board
CERNOpto-
Rx Analogue/Digital
96 TrackerOpto Fibres
VMEInterface
XilinxVirtex-IIFPGA
Data Rates
9U VME64x Form Factor
Modularity matched Opto Links
Analogue: 96 ADC channels (10-bit @ 40 MHz )
@ L1 Trigger : processes 25K MUXed silicon strips / FED
FPGAFPGA Digital Processing
Raw Input: 3 Gbytes/sec*
after Zero Suppression...
DAQ Output: ~ 200 MBytes/sec
~440 FEDs required for entire SST Readout System
FE-FPGAClusterFinder
TTC
TCS
TempMonitor
JTAG
TCS : Trigger Control System
9U VME64x