Start-up & synchronization sequence for Front-End LHCb Electronics Upgrade Meeting 13 February 2014...

Start-up & synchronization sequence for Front-End

LHCb Electronics Upgrade Meeting13 February 2014

F. Alessio, CERN

LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 2

Scope & Outline

Aim at describing the way in which the system would globally:

- Synchronize the readout of events at the beginning of a run («Start-of-run fast sequence»)

o to ensure TELL40 code is able to align to incoming data stream

o to ensure FE is aligned to correct BXID

- Resynchronization mechanisms• After a desynchronization• Preventive resynch to avoid loss of data• Usage of FE Reset and Synch command


Synch and alignment of TFC links

TELL40sTELL40s

Front-EndsFront-Ends

SOL40S-ODIN TELL40s

Front-Ends

LHC Clocks

= Receiver

= Transmitter

GBT for TFC+ECS GBT for

DATA

FE ASIC

FE ASIC

FE ASIC

1. Synch and align the TFC links: send Synch pattern in 8b/10b

BXID(12 bits)

Synch Pattern (h bits) ...00000...

630

Firmware will check synchronization of buffer and alignment of content of frame When done, set synch bit via ECS.


TELL40sTELL40s


SOL40S-ODIN TELL40s

Front-Ends

LHC Clocks

= Receiver

= Transmitter


DATA

FE ASIC

FE ASIC

FE ASIC

2. Synch and align TFC-TELL40 links: send Synch pattern in 8b/10b

BXID(12 bits)


630


Firmware will check synchronization of buffer and alignment of content of frame When done, set synch bit via ECS.


TELL40sTELL40s


SOL40S-ODIN TELL40s

Front-Ends

LHC Clocks

= Receiver

= Transmitter


DATA

FE ASIC

FE ASIC

FE ASIC

3. Synch and align TFC-FE links: send TFC commands in GBT according to each FE’s mapping

SOL40 must be configured before FE and clock transmission must be up (minimal SODIN configuration)


GBT does the job. When synch’d, can start configuring FE.


TELL40sTELL40s


SOL40S-ODIN TELL40s

Front-Ends

LHC Clocks

= Receiver

= Transmitter


DATA

FE ASIC

FE ASIC

FE ASIC

3bis. Special mode: relay TFC commands back to SOL40 measure latency


SOL40 will measure transmission delay by comparing BXID Interested only in

BXID latency, not fine phase

However if at the limit might see some desynch and adjust...

LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN

BXID(12 bits)


79 or 1110

7

TELL40sTELL40s


SOL40S-ODIN TELL40s

Front-Ends

LHC Clocks

= Receiver

= Transmitter


DATA

FE ASIC

FE ASIC

FE ASIC

4. Synch and align FE-TELL40 links: send Synch Pattern in GBT


TELL40 LLI (GBT decoding+ GX buffer) will synchronize the links Special Synch Pattern is used to align the data stream to TELL40 processing logic


TELL40sTELL40s


SOL40S-ODIN TELL40s

Front-Ends

LHC Clocks

= Receiver

= Transmitter


DATA

FE ASIC

FE ASIC

FE ASIC

4bis. Special mode: relay TFC commands back to TELL40 measure latency


TELL40 will measure transmission delay by comparing BXID received by SOL40 and FE Interested only in

BXID latency, not fine phase

However if at the limit might see some desynch and adjust...


ECS start-of-run sequence

Just like now, current LHCb experiment

CONFIGU

RE

• SOL40s get configured first, then FE gets configured, via SOL40.• Everything goes “READY” (if ok…) (very quickly…)• Links are synch’d across the whole system

READY

• FE is free running in this state (idle state at FE), but ready to react on TFC commands. On output link send random frames to keep GBT link up

• ECS issues “RUN”. Some configuration might go via SOL40 to FE (if)

RUNNIN

G

• Everything is ready to take data, SODIN is still in ACTIVE. ECS issues “GO”• Fast TFC start-of-run sequence starts. Then, data taking!


TFC start-of-run fast sequence


TFC start-of-run sequence

FE Reset is the first thing issued via control system: asynchronous!

Issued when TFC receives command «GO»

Ensure that FE got the right BXID value before synch-ing to TELL40

This sequence is the same every time a FE Reset is issued. If FE Reset is ussed, change Run #. For «in-Run» resynch, use Synch

command.


What to do on SYNCH?

Synch command is meant to be sure that (whole) system is synchronized… in a synchronous way!

FEs send Synch Patter for the same BXIDs everywhere• FE frees its memory : delete its content, read and write pointers back to empty• FE sends Synch Pattern for as many as Synch commands are set by TFC• Next event, starts at bit 0 (LSB) – might be header only• TELL40 will align to corresponding frame and BXID

TELL40 closes all events before and sends them out truncated TELL40 does not know what BXID to expect. Synch pattern says it.

Note: Synch command can be used on lab tests without TFC. For few links, system can align independently of TFC (just program your FE to send it and TELL40 to receive it). TELL40 would be in pass-through (i.e. accept all).

As soon as you need synchronicity across all links, then you need TFC. TFC will ensure that synch command is sent out for same BXID everywhere automatically aligning TELL40 readout.

Send something like this! BXID

(12 bits)Synch Pattern (h bits) ...00000...

79 or 1110


What to do on SYNCH?

Double usage (in AND or in OR):

1. Periodically: i.e., SYNCH command sent every n Hz a la FE Reset, but without resetting a bit inefficient to clear the FE buffer (could be like this only at the

beginning and when very few bunches – lots of empty-empty crossing)

2. Asynchronously: i.e. when a desynch is detected, like TELL40 detects wrong frames, wrong packing

needs fast diagnostics in TELL40 codes makes sense to clear the FE buffer in this case

could be sent only for a local sub-detector from SOL40 • (slow) triggered by ECS (TELL40 or FE set a desynch bit, ECS sends

command to SOL40) • or by TELL40 via SOL40 transmitting an info field regarding this w/

throttle protocol (we have plenty of BW)

Resynchronization sequence

Send something like this! BXID

(12 bits)Synch Pattern (h bits) ...00000...

79 or 1110


Resynchronzation sequence

Synch command was requested From ECS or fast via TELL40

Note: Synch command is entirely programmable in frequency, length and location around the orbit. Header Only is entirely programmable in length.


Conclusion

TFC documents updated, please review and comment.

LHCb-PUB-2012-001 (TFC specs)LHCb-PUB-2012-017 (FE and BE specs)

https://cds.cern.ch/record/1424363?ln=en

https://cds.cern.ch/record/1491666


Backup


System and functional requirements

1. Bidirectional communication network

2. Clock jitter, and phase and latency control

At the FE, but also at TELL40 and between S-TFC boards

3. Partitioning to allow running with any ensemble and parallel partitions

4. LHC interfaces

5. Events rate control

6. Low-Level-Trigger input

7. Support for old TTC-based distribution system

8. Destination control for the event packets

9. Sub-detectors calibration triggers

10.S-ODIN data bank

Infomation about transmitted events

11.Test-bench support


The S-TFC system at a glance

DATA

DATA

S-ODIN responsible for controlling upgraded readout system • Distributing timing and synchronous commands• Manages the dispatching of events to the EFF• Rate regulates the system• Support old TTC system: hybrid system!

SOL40 responsible for interfacing FE+TELL40 slice to S-ODIN• Fan-out TFC information to TELL40• Fan-in THROTTLE information from

TELL40• Distributes TFC information to FE • Distributes ECS configuration data to FE• Receives ECS monitoring data from FE

STORAGEReadout

Supervisor S-ODIN

Interface boardsSOL40

Readout BoardsTELL40s

Front-Ends

LHC Interfaces

TFC

TFC + ECS

TFC

FARMECS (FE)

LLT

THROTTLEECS

40 Gb/s

3.2 Gb/s

4.8 Gb/s

3.2 Gb/s

1 Gb/s

2.4 Gb/s 40 MHz clock


S-TFC concept reminder

19

LHC Interface

Clock Fanout

CLK

ECS

PH

Y

Programmable Switch layer (Partitioning)

Built-in GX Transceivers layer

PH

Y FARM

n

n

#links = #LHCb sub-systems

S-ODIN

TFC+ECS Interface

TF

C,

Th

rott

le

FAN-OUT/FAN-IN Logic

+ECS to FE

encoding Logic+

optional S-ODIN Logic

Switch Logic

Master FPGA (STRATIX IV/V GX)

S-ODIN Logic Instantiations (x6)

Master FPGA (STRATIX IV/V GX)

LLT

FE electronics

TELL40s

GB

T

FPGA-to-FPGA transceivers

S-ODIN Logic

PH

Y

ECS to FE


Readout Crate

TE

LL

40s

FEs FEs FEs...FEs FEs FEs...FEs FEs FEs...

SO

L40

TFC on backplane

ECS

TFC Crate

S-O

DIN

, LL

T, L

HC

LHC clock, LHC interfaces

TR

IG40

s

FARM

DATATFC+ECS

TE

LL

40s

TE

LL

40s

TE

LL

40s

TE

LL

40s

TE

LL

40s

TE

LL

40s

TE

LL

40s

The upgraded physical readout slice - ATCA

Common electronics board for upgraded readout system: AMC40 cards fitted in an ATCA motherboard

• S-ODIN & SOL40 AMC cards• LLT & TRIG40 AMC cards• TELL40s AMC card• LHC Interfaces specific AMC card

20


The upgraded physical readout slice – PCIe

Common electronics board for upgraded readout system: PCIe-Gen3 card fitted on a host PC • SODIN & SOL40 PCIe card• LLT PCIe ard• TELL40 PCIe card• LHC Interfaces specific PCIe card

21


TFC protocol to TELL40

22

«Extended» TFC word to TELL40 via SOL40: 64 bits sent every 40 MHz = 2.56 Gb/s packed with 8b/10b protocol (i.e. total of 80 bits) no dedicated GBT buffer, use ALTERA GX simple 8b/10b

encoder/decoder

THROTTLE information from each TELL40 to SOL40: • 1 bit for each AMC board + BXID for which the throttle was set

merged and aligned in SOL40 same GX buffer as before (same bidirectional transceiver)

Constant latency after BXID

BXID(11..0) MEP Dest(31..0) Trigger Type(3..0) Calibration Type(3..0)

Trigger BX Veto NZS Mode

Header Only

BE Reset

BXID Reset

FE Reset

EID Reset

Synch Snapshot

Reserve

0123456789

13 .. 1017 .. 1449 .. 18505163 .. 52MEP

Accept

MEP accept command when MEP ready: Take MEP address and pack to FARM No need for special address, dynamic


TFC protocol to FE

23

TFC word on downlink to FE via SOL40 embedded in GBT word: 24 bits in each GBT frame every 40 MHz = 0.98 Gb/s all commands associated to BXID in TFC word

Put local configurable delays for each TFC command • GBT does not support individual delays for each line• Need for «local» pipelining: detector delays+cables+operational logic (i.e. laser

pulse?) DATA SHOULD BE TAGGED WITH THE CROSSING TO WHICH IT BELONGS!

TFC word will arrive before the actual event takes place• To allow use of commands/resets for particular BXID• Accounting of delays in SODIN: for now, 16 clock cycles earlier + time to receive• Aligned to the furthest FE (simulation, then in situ calibration!)

TFC protocol to FE has implications on GBT configuration and ECS to/from FE• see specs document!

23 .. 12 11 10 9 8 .. 5 4 3 2 1 0

BXID(11..0) Synch NZS Mode

Header Only

BXID Reset

FE ResetCalibration Type(3..0) BX VetoSnapshotReserve


SODIN firmware v1r0 – block diagram

OR

BIT

BXID cnt BXIDBCLK

Internal Triggers

RndTrg Gen

PerTrg Gen

PerTrg Gen

CalibTrg Gen

CalibTrg Gen

CalibTrg Gen

CalibTrg Gen

BXID

RndTrgA

LumiTrgs

RndTrgC

RndTrgD

PerTrg01

PerTrg02

CalibTrgA

CalibTrgB

CalibTrgC

CalibTrgD

LLT INPUT LLT Trg

LLT Type

AUX INPUTAUX Trg

Trg Manager

FE reset handler

BE reset handler

FE RESET (from ECS)

BE RESET (from ECS)

Header Only TRG

BX Veto TRG

Trigger TRG

Trigger Type TRG

LLT Trg Enb

LLT Beamgas Enb

AUX Trg Enb

Calib Type TRG

Trigger BXID TRG

NZS/TAE handler

LLT BXID

BXID

NZ

S e

nb

NZ

S

con

secu

tive

NZ

S w

ait

EID RESET (from ECS)

EID reset handler

TA

E e

nb

FE Reset TRG (1 clock cycle)

BE Reset TRG (1 clock cycle)

Header Only TAE

BX Veto TAE

Calib Type TAE

Trigger BXID TAE

NZS mode TAE

Synch(from ECS)

Synch cmd

handler

Syn

ch Cm

d

MEP handler

FARM

ME

P w

ind

ow

MEP destination

TFC WORD (64bits in 8b/10b

encoder) ~2.5Gb/s

ME

P

des

tinat

ion

enb

Throttle handler

Throttle

Throttle BXID

BXID

S-ODIN Master Logic

SODIN data bank creator SODIN data bank to FARM

(12 x 32b Ethernet)~16Gb/s

ME

P d

estinatio

n

Trg BX Type TRG Trg BX Type TAE

UTC start time(from ECS)

UTC time++

OD

IN d

ata

ban

k e

nb

Throttle enb

BX Type Handler BX Type

WriteRead

Fill

ing

S

che

me

(f

rom

EC

S)

Filling Scheme RAM (224 x 32b)

EID cntEID accept

Run number (from ECS)

EID Reset TRG (1 clock cycle)

Synch Cmd TRG (2 or more clock cycles)

TCK (from ECS)

Snapshot(from ECS)

Snapshot cmd

handler

Synch length(from ECS)

Snapshot interval

(from ECS)

BX

ID R

ese

t

FE Reset TAE

BE Reset TAE

EID Reset TAE

Synch Cmd TAE

TA

E la

tenc

y co

mp

ens

atio

n

Trigger Type TAE

Trigger TAE

FE

Re

set wa

term

ark

FE

Re

set

BE

Re

set w

ate

rma

rk

BE

Re

set

EID

Re

set

Th

rottle

NZ

S+

TA

E

wa

term

ark

TA

E w

ind

ow

EID

Re

set

Sn

apsh

ot C

md

BXID reset handlerBXID

PA

US

E (fro

m E

CS

)

ME

P d

yna

mic

d

estin

atio

n en

b

MEP throttle

EID


Timing distribution

25

From TFC point of view, ensured constant: LATENCY: Alignment with BXID FINE PHASE: Alignment with best sampling point

TELL40sTELL40s


TFC+ECSInterfaceS-ODIN TELL40s

Front-Ends

LHC Clocks

= Receiver

= Transmitter


DATA

FE ASIC

FE ASIC

FE ASIC

Some resynchronization mechanisms envisaged: Within TFC boards With GBT

No impact on FE itself

Loopback mechanism: re-transmit TFC word

back allows for latency

measurement + monitoring of TFC commands and synchronization


DCS objectDCS

object

TELL40TELL40TELL40

GBTX SCA SCA

DCS object

GBTX GBTX GBTX GBTX GBTX GBTX

FE ASIC

FE ASIC

FE ASIC

FE ASIC

FE ASIC

24

DATA

TFC commands

SOL40

SCA

TFC+ECS GBT

DATA GBT

SCA SCA SCA Configuration data / monitoring data

I2C, JTAG…

Clock

TFC+ECS

Generic FE electronics architecture26

How to decode TFC in FE chips?

Usage of TFC+ECS GBTs in FE is 100% common to everybody!! dashed lines indicate the detector

specific interface parts please pay particular care in the

clock transmission: the TFC clock must be used by FE to transmit data, i.e. low jitter!

Kapton cable, crate, copper between FE ASICs and GBTX

FE electronics block


FEModule

FEModule

Phase – Aligners + Ser/Des for E – Ports

FEModule

E – PortE – Port

E – Port

GBT – SCA

E – Port

Phase - Shifter



CDR

DEC/D

SCR

SER

SCR/ENC

I2C MasterI2C Slave

Control Logic Configuration(e-Fuses + reg-Bank)

Clock[7:0]

CLK Manager

CLK Reference/xPLL

External clock reference

clockscontroldata

one 80 Mb/s port

I2C port

I2C (light)

JTAG

80, 160 and 320 Mb/s ports

GBTIA

GBLD

GBTXe-Link

clock

data-up

data-down

ePLLTxePLLR

x

JTAG port

27

The TFC+ECS Master GBT

These clocks should be the main clocks for the FE• 8 programmable phases • 4 programmable

frequencies (40,80,160,320 MHz)

Used to:• sample TFC bits • drive Data GBTs • drive FE

processes


The TFC+ECS GBT protocol to FE

D0D1D2D3D4ECICHEADER FEC

GBT word: 120 bits

2x16 bits16bits2x2bits4bits

24 e-links @ 80 Mb/sfor TFC

Idle: 0110Data: 0101

1 e-link @ 80 Mb/s for

GBT internal use only 1 e-link @

80 Mb/s to GBT-SCA

SCA

16 e-link @ 80 Mb/s to 16 GBT-SCAs

for ECS

16bits16bits16bits16bits

SCAs SCAs

TFC protocol has direct implications in the way in which GBT should be used everywhere• 24 e-links @ 80 Mb/s dedicated to TFC word as a baseline (effectively 48 bits)

use 80 MHz phase shifter clock to sample TFC parallel word• TFC bits are packed in GBT frame so that they all come out on the same clock edge

Modification are possible in order to satisfy sub-detector’s FE requirements

Leftover e-links dedicated to GBT-SCAs for ECS configuring and monitoring (see later)


Words come out from GBT at 80 Mb/s

D0[14..2,0]D1D2D3D4EC[0]IC[0]H[2,0] FEC[30..6,4,2,0]

D0[15..3,1]D1D2D3D4EC[1]IC[1]H[3,1] FEC[31..7,5,3,1]

24bits x TFC à grouped e-link + 1

clock line

Other purposes?

17bits x ECSà 17 GBT-SCA

17bits x ECS msb first, odd bits

lsb second, even bits

In simple words:• Odd bits of GBT protocol on rising edge of 40 MHz clock (first, msb), • Even bits of GBT protocol on falling edge of 40 MHz clock (second,

lsb)


TFC decoding at FE after GBT

D0[1] D0[0] D0[1] D0[0]

D0[3] D0[2] D0[3] D0[2]

...... ......

D1[1] D1[0] D1[1] D1[0]

D0[7] D0[6] D0[7] D0[6]

...... ......

D2[1] D2[0] D2[1] D2[0]

D1[7] D1[6] D1[7] D1[6]

...... ......

D2[7] D2[6] D2[7] D2[6]

TFC[23..0] TFC[23..0]unused unused

24 2424 24

80 MHz

40 MHz

E-link 1, group 1

E-link 2, group 1

E-link 8, group 1

E-link 1, group 2

E-link 8, group 2

E-link 1, group 3

E-link 8, group 3

This is crucial!!

we can already specify where each TFC bit will come out on the GBT chip

this is the only way in which FE designers still have minimal freedom with GBT chip

if TFC info was packed to come out on only 12 e-links (first odd then even), then decoding in FE ASIC would be mandatory!

which would mean that the GBT bus would have to go to each FE ASIC for decoding of TFC command

there is also the idea to repeat the TFC bits on even and odd bits in TFC protocol

would that help? FE could tie logical blocks directly on GBT pins… Or to select a minimal set of TFC commands and

repeat them to profit from fan-out possibilities


Now, what about the ECS part?

Each pair of bit from ECS field inside GBT can go to a GBT-SCA • One GBT-SCA is needed to configure the Data GBTs (EC one for example?)• The rest can go to either FE ASICs or DCS objects (temperature, pressure) via other

GBT-SCAs GBT-SCA chip has already everything for us: interfaces, e-links ports ..

No reason to go for something different! However, «silicon for SCA will come later than silicon for GBTX»…

We need something while we wait for it! FPGA emulator (working on it)

D0D1D2D3D4ECICHEADER FEC

GBT word: 120 bits

2x16 bits16bits2x2bits4bits

24 e-links @ 80 Mb/sfor TFC

Idle: 0110Data: 0101

1 e-link @ 80 Mb/s for

GBT internal use only 1 e-link @

80 Mb/s to GBT-SCA

SCA

16 e-link @ 80 Mb/s to 16 GBT-SCAs

for ECS

16bits16bits16bits16bits

SCAs SCAs


Protocol drivers build GBT-SCA packets with addressing scheme and bus type for associated GBT-SCA user busses to selected FE chip Basically each block will build one of

the GBT-SCA supported protocols

Memory Map with internal addressing scheme for GBT-SCA chips + FE chips addressing, e-link addressing and bus type: content of memory loaded from ECS

SOL40 firmware

E-LinkProtcolDrivers


PCIe Memory

Map

TFC Relay &

Alignment

TFC decoder

Synchronous TFC Info to

TELL40s (64 bits)

S-ODIN synchronousTFC Info fan-out

ECS

FE


CCPCPCIe Slave

24 bits

34 bits

GBT Tx

EC

ST

FC

LHCb Electronics Upgrade Meeting, 26/07/12 F. Alessio, R. Jacobsson

Usual considerations …

33

TFC+ECSInterface has the ECS load of an entire FE cluster for configurating and monitoring 34bits @ 40 MHz = 1.36Gb/s on single GBT link

• ~180 Gb/s for full SOL40 (132 links)• Single CCPC might become bottleneck… Clara & us, December 2011

How long to configure FE cluster? how many bits / FE? how many FEs/ GBT link? how many FEs / TFC+ECSInterface?

Numbers to be pinned down soon + GBT-SCA interfaces and protocols.

Readout CrateT

ELL

40

TE

LL40

TE

LL40

TE

LL40

TE

LL40

TE

LL40

TE

LL40

TE

LL40

FEs FEs FEsFEs...FEs FEs FEsFEs...FEs FEs FEsFEs...

SO

L40

ECS

https://indico.cern.ch/conferenceDisplay.py?confId=119031






Old TTC system support andrunning two systems in parallel

34

We already suggested the idea of a hybrid system:reminder: L0 electronics relying on TTC protocol part of the system runs with old TTC system part of the system runs with the new architecture

How?

1. Need connection between S-ODIN and ODIN (bidirectional) use dedicated RTM board on S-ODIN ATCA card

2. In an early commissioning phase ODIN is the master, S-ODIN is the slave S-ODIN task would be to distribute new commands to new FE, to new

TELL40s, and run processes in parallel to ODIN ODIN tasks are the ones today + S-ODIN controls the upgraded part

In this configuration, upgraded slice will run at 40 MHz, but positive triggers will come only at maximum 1.1MHz…

• Great testbench for development + tests + apprenticeship…• Bi-product: improve LHCb physics programme in 2015-2018…

3. In the final system, S-ODIN is the master, ODIN is the slave ODIN task is only to interface the L0 electronics path to S-ODIN

and toprovide clock resets on old TTC protocol


SODIN on Marseille’s ATCA board

35

ATCA motherboard

FPGA

RJ45

CCPC GbE

Serial crossbar

PCIe

Clock

Int. Comm.

S-ODIN

AMC

AMC

LLT Trigger

AMC

LHC Interfaces

AMC

FPGA

FPGA

FPGA

FPGA

Bac

kpla

ne c

onne

ctor

Clock

TFC+ Throttles

LHC Interfaces

LLT sub-trigger

FARM

SOL40

ATCA RTM

RT

M

conn

ecto

r

Ext

erna

l co

nnec

tors

à

OD

IN

Clock crossbar


TFC+ECSInterface on Marseille’s ATCA board

36

SOL40

RJ45

CCPC GbE

FPGA

Serial crossbar

PCIe

Clock

TFC + Throttles

FE

AMC

FE

AMC

FE

AMC

S-ODIN + FE

AMC

FPGA

FPGA

FPGA

MasterFPGA

Bac

kpla

ne c

onne

ctor

Clock

TFC+ Throttles

S-ODIN

FE (GBT, opt.)

FE (GBT)

FE (GBT)

FE (GBT)

Clock crossbar


Reminder: your (generic) FE

For details, see LHCb-INT-2011-011

Compress (zero-suppress) data already at the FE• reduce # of links from ~80000 to ~12500 (~20 MCHF to ~3.1 MCHF)• data driven readout (asynchronous) + variable latencies!

Efficiently use data link bandwidth• pack data on data link continuously with elastic buffer• extensive use of GBT (robust FEC vs WideBus mode)

evaluate choices based on complexity vs robustness

NO TRIGGER to FE! Only commands, clock

and slow control

https://cds.cern.ch/record/1340939?ln=en


Fast & Slow Control to FE

Separate links between controls and data

• A lot of data to collect

• Controls can be fanned-out (especially fast control)

Compact links merging Timing, Fast and Clock (TFC) and Slow Control (ECS).

• Extensive use of GBT as Master GBT to drive Data GBT (especially for clock)

• Extensive use of GBT-SCA for FE configuration and monitoring

On detector

Off detector

4.8 Gb/s

4.8 Gb/s

TFC

ECS

Data

TFC

ECS

Data

4.8 Gb/s

Off detector


Reminder: generic FE data flow scheme

Compression/suppression logic can

have dynamic or static latency

Applies changes to data

FE buffer for data

Tag data with TFC commands and pipe them across

compresson/suppression logic block

Modify data according to TFC commands + BufferFull then pack continuously onto

GBT

Data available needed only if compression / suppression is

dynamic


The code: FE data generator

DERANDOMIZING BUFFERProgrammable Depth

Width: (number of channels * channel size) + header size

CHANNEL DATA GENERATOR

DATA OCCUPANCY GENERATOR

(POISSON)

LHC MACHINE (FILLING SCHEME)

DATA WORD(number of channels * channel size)

EVENT HEADER( 12 bits BCLK identifier

+ 4 status bits + 8 bits for data size )

ELSE

IF DATA_SIZE = 0 (no hit)or BX_VETO = 1

or HEADER_ONLY = 1

DO NOT WRITE TO DERANDOMIZER WHILE PACKING JUST HEADERS!à RECUPERATE BUFFER SPACE

à FORCE DATA SIZE TO 0 IN HEADER FOR TELL40 DECODINGà MAX # OF HEADERS IN WORD =

ROUND_TO_INT(GBT WORD SIZE / HEADER SIZE)

GBT PACKING LOGIC

GBT FRAME – EXAMPLE WITH GBT SIZE 84 BITS WITH FEC CORRECTION

IF NZS_MODE = 1 occupancy 100%

à put all channels in word

WRITE TO DERANDOMIZER

Padded to 0s... EVENT HEADER

EVENT HEADER

EVENT HEADER

...

0x5 NB: GBT HEADER = X”5” if data or X”6" if IDLEEV 01 HEADERDATA WORD of EV 01

0x5EV 02 HEADER REST OF DATA WORD of EV 01 DW of EV 02

0x5REST OF DATA WORD of EV 02

0x5REST OF EV 04

HEADER

EX: EV02 is NZS



FOLLOWING 3 EVENTS HAVE HEADER ONLY = 1 FROM TFC

EV 05 HEADEREV 06 HEADERDW OF EV 06

0x5REST OF DATA WORD of EV 06 NEXT EVENT IS IN LINE




0x5

EV 09 HEADER

DATA WORD of EV 09

.

.

.GBT ENCODER

IF FE_RESET = 1 or SYNCH = 1

à reset derandomizer buffer

0x5EV XX HEADER00000MANUAL ALIGNMENT FRAME

(programmable via ECS)

.

.

.

This part should be replaced by sub-detector’s specific data generated from MonteCarlo simulation

NB: data word width can include other informationà e.g. width would be (number_of_channels *

channel_size) + hit_pattern_size (== number_of_channels)

0x600000DERANDOMIZER IS EMPTY, SEND IDLE

FRAME OVER GBT FRAME

0x5... TWO EMPTY EVENTS

DERANDOMIZER ALMOST FULL

à leaving some buffer space for safety margin

SYNCH will always follow a FE RESET in order to achieve link synchronizationà during SYNCH command asserted, FE should exchange data field with MANUAL

ALIGNMENT FRAME for links synchronization

0x5EV XX+1 HEADER

00000MANUAL ALIGNMENT FRAME


READ FROM DERANDOMIZER

FIXED OR DYNAMIC LATENCY,

Not implemented and sub-detector specific

SUPPRESSION / COMPRESSION

MECHANISM


à SET TRUNCATED BIT = 1à MAX # OF HEADERS IN WORD =



EVENT HEADER

EVENT HEADER

...

EV 03 HEADEREV 04

HEADER

GBT ENCODER

ELSE

EV 10 HEADEREV 11

HEADER

REST of EV 11 HEADER


The code: FE buffer manager

DERANDOMIZING BUFFERProgrammable Depth



DATA OCCUPANCY GENERATOR

(POISSON)





ELSE


or HEADER_ONLY = 1




GBT PACKING LOGIC






EVENT HEADER

EVENT HEADER

...




0x5REST OF EV 04

HEADER

EX: EV02 is NZS









0x5

EV 09 HEADER

DATA WORD of EV 09

.

.

.GBT ENCODER





.

.

.




0x600000 DERANDOMIZER IS EMPTY, SEND IDLE FRAME OVER GBT FRAME






0x5EV XX+1 HEADER







MECHANISM





EVENT HEADER

EVENT HEADER

...

EV 03 HEADEREV 04

HEADER

GBT ENCODER

ELSE

EV 10 HEADEREV 11

HEADER



The code: GBT dynamic packing DERANDOMIZING BUFFERProgrammable Depth



DATA OCCUPANCY GENERATOR (POISSON)





ELSE


or HEADER_ONLY = 1




GBT PACKING LOGIC






EVENT HEADER

EVENT HEADER

...




0x5REST OF EV 04

HEADER

EX: EV02 is NZS









0x5

EV 09 HEADER

DATA WORD of EV 09

.

.

.GBT ENCODER





.

.

.




0x600000 DERANDOMIZER IS EMPTY, SEND IDLE FRAME OVER GBT FRAME






0x5EV XX+1 HEADER







MECHANISM





EVENT HEADER

EVENT HEADER

...

EV 03 HEADEREV 04

HEADER

GBT ENCODER

ELSE

EV 10 HEADEREV 11

HEADER


Very important to analyze simulation

output bit-by-bit and clock-by-clock!


Reminder: dynamic packing algorithm

01234

Average event size =

link bandwidth

Buff

er d

epth

Average event size

01234

Link bandwidth

01234

BX0BX1BX2BX3BX4

BX0BX1

BX2 BX3 BX4

Header is the unique identifier for each event in frame Compulsory (tag for each crossing), partly programmable (must contain length of

frame+BXID) Difficult buffer management, but almost no truncation. Flexible against occupancy problem (what if your estimate is wrong?). Maximum exploitation of bandwidth. Readout Board uses Header to decode and separate frames lots of resources.

+ =


Reminder: fixed packing algorithm

01234

Average event size /=link bandwidth Truncation!

Buff

er d

epth

Average event size

01234

Link bandwidth

01234

BX0BX1BX2BX3BX4

BX0BX1BX2BX3BX4

This is different: one clock cycle one event one GBT frame Header more flexible: you can add addresses, hitmaps… Very simple buffer management, but truncation has to happen eventually. Not flexible against occupancy problem (again, what if your estimate is wrong?). Loses a bit of bandwidth as empty spaces must be padded to be sent out. Readout Board uses Header to decode and separate frames much fewer resources

+ =


Reminder: fixed vs variable length header

in dynamic packingDynamic packing with fixed length header.

Dynamic packing with dynamic length header (fully flexible!)


Few comments to start with:- BX Veto and Header Only commands are identical from FE point of view

ORed- TFC commands are synchronous wrt to BXID Reset

once we align BXID Reset with beam, TFC commands come ALWAYS at the same latency

• (wrt to BXID Reset, hence BXID)! Compression/suppression logic should act accordingly to TFC command

• (why would you want to compress/suppress if that crossing is rejected a priori? Especially if your pre-processing is dynamic…)

- Data is filtered according to TFC commands and the FE buffer status- Data is packed onto the GBT link in a continuous fashion

FE flow control scheme

BX Veto with a calib trigger in between

Clock

FE Reset

Header Only

Other cmds

BXID Reset

BX Veto

Calib type[0]

Length of BX veto depends exclusively on filling scheme

Fixed distance after alignment


Data Valid

GBT can accept DATA or IDLE frame: Send IDLE frame whenever a GBT frame

is not ready to be sent! IDLE frame can contain whatever your

sub-detector wants to send.

See TELL40 fw specs, coming soon…

Data Valid signal to distinguish between DATA and IDLE frame:

Clock 80 MHz

Data Valid

Data

Data Valid Usage

Clock 40 MHz

IDLE DATA DATA DATA DATA DATAIDLE


Data Valid

Be careful to rise synchronize the Data Valid signal to the right rising edge when using the 80 MHz clock (or 160 or 320…)

Clock 80 MHz

Data Valid

Data

Data Valid Wrong Usage

Clock 40 MHz


GBTX would split the frame in this case!!Synchronize your DV signal to the beginning of the GBT frame!


Data Valid

Some sub-detectors will connect more FEs to the same GBT transmitter:- Each FE with its own memory- Can happen that one can send DATA, the other cannot! (IDLE vs DATA in the same

packet)

Keep DV always high!

Clock 80 MHz

Data Valid

Data

Clock 40 MHz


Data Valid Many Channels

DATA DATA DATA DATA DATAIDLE

You HAVE to indicate whether the packet was DATA or IDLE, by sacrificing one bit your DATA/IDLE frame


Data Valid

Some sub-detectors will connect more FEs to the same GBT transmitter:- Each FE with its own memory- Can happen that one can send DATA, the other cannot! (IDLE vs DATA in the same

packet)

Keep DV always high!

Clock 80 MHz

Data Valid

Data

Clock 40 MHz


Data Valid Many Channels

DATA DATA DATA DATA DATAIDLE

You HAVE to indicate whether the packet was DATA or IDLE, by sacrificing one bit your DATA/IDLE frame

Date post:	29-Jan-2016
Category:	Documents
Upload:	patience-simon
View:	226 times
Download:	0 times

Start-up & synchronization sequence for Front-End LHCb Electronics Upgrade Meeting 13 February 2014...

Documents