1
FPGA firmware of DC5 FEE
Chih-Hsun Lin, Yu-Sheng Teng, Ming-Lee Chu,Chia-Yu Hsieh, Takahiro Sawada, Wen-Chen Chang
Institute of Physics, Academia Sinica, Taiwan
Tobias Grussenmeyer, Horst FischerInstitute of Physics, University of Freiburg, Germany
DC5 FPGA firmware review 2014/5/15
2
Outline
• System architecture– Module connection– Signal/CLK connection
• FPGA design– FEM
• TDC measurement• Trigger match
– DCM• Data packing and transfer
• List of identified problems during the on-site test• Summary
FEM-DCM Architecture
3
• FPGA in FEM (LFXP2-8E-6FT256C)• 12 EBR Blocks (221184 bits)
• FPGA in DCM (Spartan-6 xc6slx45t-3fgg484)• 116 18Kb RAM Blocks = 232 9Kb RAM block
4
Module Connection
Signal and Clock rate
— Optical Tx and Rx 3.1104 Gbps— Master CLK 38.88 MHz (was 155.52MHz)— Command 38.88 Mbps (was 155.52MHz)— FLT (First level Trigger) Pulse sync to 38.88MHz CLK — Data 155.52 Mbps
Command & FLTGANDALF DCM FEM
Optical fiberEthernet
cable
Master CLK
Data
FEM
……
Optical Tx
Optical Rx
x1 x8 x20
Totally1 GANDALF8 DCM160 FEM
5
Experimental Parameters
• Highest hit rate per wire = 200 kHz• Noise rate per wire = 10 kHz (conservative estimation based
on the measured value at 4 fC threshold)• Overall hit occupancy = 10 %• Trigger matching window = 200 ns• Highest instantaneous trigger rate = 100 kHz, i.e. minimum
time between two consecutive triggers = 10 μs.• Trigger latency = 8 μs
• We will use these parameters to evaluate the proper specifications of FEE.
6
Data Transfer• FEM DCM
– Data format : 32 bits- Header : 2 bits- FEM ID (0-19): 6 bits- channel ID (0-15): 4 bits- Time (range 70 s, finest 1-ns resolution): 16 bits- PLL locked: 4 bits
- Speed = 155.52 Mbps - Maximum transmission time per trigger = 32bits/word * (16+2) words / 155.52 Mbps
~ 576 bits/ 155.52Mbps ~ 3.7 μs < 10 μs trigger period – Minimum cyclic buffer size : (200+10) kHz *16 hits * 8μs ~ 30 hits < 512 hits buffer
size
• DCM GANDALF- Speed = 3.11 Gbps - Maximum transmission time per trigger = 32bits/word * [(16+2) words *20 FEM +
5 S-link header words]/ 3.11 Gbps
~ 11680 bits/ 3.11 Gbps ~ 3.8 μs < 10 μs trigger period
• GANDALF ROB- Speed : 160 MByte/s = 1.28 Gbps- Maximum transmission time per trigger = 32bits/word * [(16+2) words *20 FEM +
5 S-link header words]* 8 DCM *10% occupancy/ 1.28 Gbps
~ 9344 bits/ 1.28 Gbps ~ 7.3 μs < 10 μs trigger period
7
Clock Structure
GANDALF
DCM FEM
Optical Tx
Optical RxXilinx
Transceiver
OSC155.52M
Rx clock155.52M
AlignCommaK28.1 k28.5
Deformater PLL1
PLL2
155.52M
FEM Ctrl Logic
77.76M
FEM2DCMtransmit
155.52MB
TDC
Clock 38.88M
233.28M
233.28M 90°
2M CMAD setting
• Deformater is a VHDL code from T. Grussenmeyer.— Master 38.88MHz CLK is generated and phase adjusted by RX CLK and
command from GANDALF.• Two TDC CLKs will be 233.28MHz = 38.88MHz x 6. (248MHz now)• 155.52MB is a phase adjustable CLK for data output.
8
First Level Trigger
• In DCM, First level trigger is generated according to a specific command/data pattern from GANDALF.
• A FLT pulse is distributed to FEMs sync to master CLK.
GANDALFOptical Tx
Optical RxXilinx
Transceiver
Rx clock155.52M
FLTCommaK28.0
Deformater
Clock 38.88M
FLT(first level trigger)
9
TCS Reset signal
• TCS reset will be distributed to FEMs by a dedicated command sync to master CLK.– Implemented on DCM and FEM.• A command to FEMs to reset TDC counters in the
current implementation.
– Need inputs about TCS Reset command/pattern from GANDALF.
10
Trig Time
FEM TDC Block Diagram
CMADx16 inputs TDC
x16
FLT trigger TDC
x1TriggerLogic
Flag
Reset
TriggerMatch
Trig FlagEventFIFO(512 x 32bit)
DCM & FEM(8b/10b)Link logic
BufferCtrl
Logic
x16 Time
x16 Flag
x16 Reset
Cycling buffer
(512 hits)
Trig Flag
Write point
# of TDC hits
0100110110Serial data
Commandhandler
It will be4096x32 bit.
Trig Time
Data
11
TDC counter
• TDC value for each hit is 16 bit.– MSB 14 bit is from a counter by 233MHz CLK.– LSB 2 bit is determined by a four bit pattern
latched with 233MHz CLK and 233MHz 90o CLK.
14 bits 2 bits
233.28M
233.28M 90°
14 bits ≈ 70.3us
1 unit = 1.07ns
12
Trigger Match• Data and control parameters:
– TDC time = 16 bit (1ns lsb) [maximum 70 s]– Trigger Latency = 12 bit (4ns lsb) [maximum 16 s]– Matching Window = 12 bit (4ns lsb) [maximum 16 s]
• (Ttrig–Tlatency–Twindow) < Thit < (Ttrig–Tlatency+Twindow)
• Matching process stops at either one of the following three conditions:– 16 matched hits.– No more TDC hit for matching(Max hits for matching process is 255).– 4 unmatched hits after last matched hit.All the parameters for these three conditions could be adjusted and optimized according to the realistic experimental conditions.
wire
FLT
TlatencyThit Ttrig
Twindow
13
DCM block diagram
010011Serial data
DCM & FEM(8b/10b)
Transmitter
x20
DCM to FEMCommand
FIFO
x20
FEM to DCMDataFIFO
(512 x 32bit)
x20
Commandhandler
GANDALFDCM
Link logic
Transceiver 010011Serial data
TCS info
command
FLT(first level trigger)
1bit Event Counting
FIFO
x20data
packing
dataDCM & FEM
(8b/10b)Receiver
x20
010011Serial data
14
Current data-packing procedure in DCM
Idle
Any FEM framevalid
No
Wait up to 4 system clocks
Yes
Timeoutor
all FEM valid
No
Send S-link begin mark
Yes
Send S-link header
Scan FEM FIFO
framevalid
Readout FEM data
EOF wordNo
All FIFOs scaned
Yes
Yes
No
Send S-link end mark
Yes
Power on reset
No
15
List of Identified Problemsduring the on-site test
• Data loss (DCM to GANDALF)• Command error (DCM to FEM)• Command lost (PC with USB connection to GANDALF)
16
Data Loss issue(DCM to GANDALF)
• Previous version of DCM FPGA design, the same state machine controls both command flow and data flow.– Whenever a command arrives in DCM, the data packing
and transmission is interrupted. This is the found cause of the data loss.
• Solution: The DCM FPGA design is modified to have independent control for:– Command flow (GANDALF DCM FEMs)– Data flow (FEMs DCM GANDALF)
17
Latest Data transmission Test(with new DCM firmware)
2. DCM generates100k trigger in one sec.
GANDALF DCM FEM
5. Use counter check validdata frame number from FEM
Optical fibre
x1 x1 x1
Count mode
USB Ethernet cable
1. Pass command Trigger_on to DCM
3. FEM send 1 data frameto DCM per trigger
4. Packing data frame andpass to GANDALF
6. Pass data frame to PC7. Save data into fileand use program analysis
• Test result– DCM did receive 100k data frames from FEM.– There is a loss of 1.5% data frame for the data recorded in PC and the
cause is likely the current USB I/O capability. The data rate for 100k Hz trigger test is 100k*4 byte per word* 23 words per FEM = 9.2 MB/sec.
18
Command Error issue(DCM to FEM)
• A timing issue, long operational logic path due to– 8b/10b encoding– Multiplexing of commands and fill pattern
• Solution: Pipeline/FIFO is added in the DCM FPGA design to reduce logic path.– Test done. All 20 FEM ports work correctly.
19
Command lost issue(PC with USB connection to GANDALF)
• DCM could not 100% successfully receive the control command issued from PC via GANDALF if PC is reading data from GANDALF and sending commands to GANDALF simultaneously.This is due to the handshaking of USB port between PC and GANDALF. It is not a valid issue for the reality where the data control is via the VME backplane.
• Solution: Make sure that the trigger is properly stopped before sending any other control commands.
20
Summary
• The proper functioning of clock/command/data chain in the current FPGA firmware is verified and the optimization study will continue.
• All identified problems of digital transmission during the on-site test have been investigated and solved.
• We trust that the current FPGA firmware and the hardware design overall do fulfill the experimental requirements of DC5 FEE. It should be reasonable to move toward the stage of mass production in this project.
21
BACKUP
22
Command lost (DCM to FEM)
DCM to FEMCommand FIFO
• Timing issue when do 8b10b encoder– Original structure
8b10bencoder
control
command
Idle(K28.5) Trig func
Mode controlstatus
0101001100
ERROR
control
Controller + Serializer
23
Fix Command lost (DCM to FEM)
• New version
DCM to FEMCMD FIFO
CMD CMD / idleSelector
32b to 8bFIFO
8b10bencoder
Serializer
Trig modefunc
Mode select with command for DCM0101001100
FLT pulse signal
24
CMAD Control (1) Functionality Description Command Note
ThresholdSetting (Threshold , Ch_id)
Set Threshold for 1 channel
dumy = Threshold And 0x3FFByte (0) = 0x80 + (dumy And 0xF)dm = (dumy And 0x3F0) / 16Byte (1) = 0x90 + 4 + (dm And 0x1)Byte (2) = 0xA0 + (id And 0xF) Byte (3) = 0xB0 dm = (dumy And 0x3E0) / 32Byte (4) = 0x80 + (dm And 0xF)dm = (dumy And 0x200) / 512Byte (5) = 0x90 + (dm And 0x1)Byte (6) = 0xA0 + (id And 0xF)Byte (7) = 0xB0
• Control 8 bytes• Threshold 0-650 digit• Ch_id 0-8
25
CMAD Control (2) Functionality Description Command Note
BaselineSetting(Baseline,ch_id)
Set gain for 1 channel
dumy = Baseline And 0x3FF Byte(0) = 0x80 + (dumy And 0xF)dm = (dumy And 0x3F0) / 16Byte(1) = 0x90 + 6 + (dm And 0x1)Byte(2) = 0xA0 + (id And 0xF)Byte(3) = 0xB0dm = (dumy And 0x3E0) / 32Byte(4) = 0x80 + (dm And 0xF) dm = (dumy And 0x200) / 512Byte(5) = 0x90 + 2 + (dm And 0x1)Byte(6) = 0xA0 + (id And 0xF)Byte(7) = 0xB0
• Control 8 bytes• Baseline ~650 digit
26
CMAD Control (3) Functionality Description Command Note
Gain_Setting(Gain,Ch_id)
Set gain for 1 channel
dumy = Sgain And 0xF Byte(0) = 0x80 + (dumy And 0xF)Byte(1) = 0x90 + 8Byte(2) = 0xA0 + (id And 0xF)Byte(3) = 0xB0Byte(4) = 0x80 + (dumy And 0xF)Byte(5) = 0x90 + 12Byte(6) = 0xA0 + (id And 0xF)Byte(7) = 0xB0
• Control 8 bytes• Gain 0-15 digit
27
CMAD Control (4) Functionality Description Control Note.
Setting (OneShot, Ch0TestOutput, SlowRateLimit, ThresVoltageOutput)
Bit 7-4 : 0xC0
Ex. We usually set“One Shot” On +”Ch0 Test Output” Off+” Slew Rate Limit” On+”Thres Voltage Output” OffSetting(0,0,1,0)0xC0+ b0010 = 0xC2
OneShot - Advantage: Reduce “bounce” of output signal after Dis. because of noise.- Disadvantage: Decrease the sensitivity. (see backup slide)
Bit 3 On = 0Off =1( def = 1)
Ch0TestOutput - Output signals of ch0 after preampand before discriminator. - Signal of Ch0 after Dis. will be turnoff if we turn on the Ch0TestOutput.
Bit 2 On =1Off =0 ( def = 0)
SlewRateLimit - Advantage: Avoid the oscillation and hold the baseline. - Disadvantage: Decrease the bandwidth of chip. (see backup slide)
Bit 1 On = 1Off = 0 (def = 1 )
ThresVoltageOutput - Output the voltage of threshold to test pins.
- If we turn of ThresVoltageOutput, the output of all channels will be turn of.
Bit 0 On =1Off = 0 (def = 0)
28
CMAD Control(5)
SelfTest(OddCh, EvenCh)
Bit 7-4 : 0x1
Ex.Test both odd and even channels SelfTest(1,1) 0x18
Ex.Test both only odd channels SelfTest(1,0) 0x13
OddCh Output clock signals generated by FPGA to Pin4 of CMAD (Test odd channels of CMAD)
Bit 0-1 :0x3(def = 0x0)
EvenCh Output clock signals generated by FPGA to Pin6 of CMAD (Test even channels of CMAD)
Bit 2-3:0xC(def = 0x0)
Functionality Description Control Note.
29
FEMID(6)
ChID(4)
DCM ID(10)
ChID(4)
FEMID(6)
30
S-Link Header(1)
must be defined.
31
S-Link Header(2)
chosen
