LHCb Timing and Fast Control System

LHCb DAQ Review, September 11-12

LHCb Timing and Fast

Control System

TFC Team:Arek Chlopik, WarsawZbigniew Guzik, WarsawRichard Jacobsson, CERNBeat Jost, CERN

• Introduction to the TFC system

• Progress and status

2LHCb DAQ Review, September 11-12

LHCb Read-out

Read-out Network (RN)

RU RU RU

6-15 GB/s

6-15 GB/s

50 MB/sVariable latency

L2 ~10 msL3 ~200 ms

Control &

Monitoring

LA

N

Read-out units (RU)

Timing&

FastControl

Level-0

Front-End Electronics

Level-1

VDET TRACK ECAL HCAL MUON RICH

LHC-B Detector

L0

L1

Level 0Trigger

Level 1Trigger

40 MHz

1 MHz

40 kHz

Fixed latency 4.0 s

Variable latency <1 ms

Datarates

40 TB/s

1 TB/s

1 MHzFront End Links

Trigger Level 2 & 3Event Filter

SFC SFC

CPU

CPU

CPU

CPU

Sub-Farm Controllers (SFC)

Storage

Th

rott

le

Front-End Multiplexers (FEM)

Unique feature : Two levels of high-rate triggers• Level-0 (40 MHz --> 1.1 MHz Accept rate)

• Level-1 (1.1 MHz --> 40-100 kHz Accept rate)


Timing and Fast Control

TTCrx TTCrx

TTCrx

L1E

FEchipFEchipL1 buffer

ADCADCADC

ADCADCDSP

Con

trol

FEchip

TTCrx

L0E

FEchipFEchipFEchip

TTCrx

L0E

FEchipFEchipFEchip

TTCrx

L1E


ADCADCADC

ADCADCDSP

Con

trol

Clock fanoutBC and BCR

LHC clock

L0 trigger L1 trigger

Readout Supervisor Readout

Supervisor

Local trigger (optional)

L0

L1

Readout Supervisor

L0

L1

TFC switch L1 Throttle switch

L0 Throttle switch

SD1 TTCtx SD2 TTCtx SDn TTCtx L0 TTCtx L1 TTCtx

Optical couplers Optical couplers Optical couplers Optical couplers

TTCrx

L1E


ADCADCADC

ADCADCDSP

Con

trol

FEchip

TTCrx

L0E

FEchipFEchipFEchip

TTCrx

L0E

FEchipFEchipFEchip

TTCrx

L1E


ADCADCADC

ADCADCDSP

Con

trol

DAQ DAQ

Th

rott

le O

R

Th

rott

le O

R

TTC system

1717

L1 trigger system

GPS receiver

Consists of:

- RD12 TTC distribution system:

•TTCtx’s

•Tree-couplers

•TTCrx’s

- Components specific to LHCb:

•Readout Supervisors•TFC Switch•Throttle Switches•Throttle ORs


Use of TTC Timing, Trigger and Control distributed using the TTC system:

Channel A used to distribute (accept/reject signal)

• L0 trigger (40 MHz --> 1.1 MHz accept rate)

Channel B used to distribute (short broadcasts):

• L1 trigger (1.1 MHz --> 40-100 kHz accept rate)

• Bunch/Event Counter Resets

• Control commands (FE resets, calibration pulses)

Broadcast order is handled according to a priority scheme

Usage of the 6 (+2) user bits in the short broadcasts:

Bit/Broadcast 7 6 5 4 3 2 1 0

L1 Trigger 1 Trigger type EI D(2-bit) 0 0

Reset 0 1 R L1 EI D L1 FE L0 FE ECR BCR

Calibration 0 0 0 1 Pulse type 0 0

Command 0 0 R R R R 0 0


LHCb specific components Readout Supervisor “Odin”

all mastership in single module

TFC switch Clock, trigger and command distribution and support partitioning

Throttle switches (L0 & L1) and Throttle ORs Throttle feed-back


Readout Supervisor “Odin”

- Single module

- Clock distribution

LHC clock

- L0 handling & distribution

L0

- Auto-trigger generator

Trigger generator

- Trigger controller

Trigger controller

Throttles

- Reset/cmd generator

Reset/commandgenerator

- “RS Front-End”RS Front-End

L0/L1

DAQ- ECS interface

ECS interface

ECS

TTC encoder

Ch A/B

- TTC encoding

- L1 handling & distribution

L1

- L1 derandomizationL1

Derandomizer

Designed with emphasis on:• Versatility - to support many different types of running modes

• Functionality easily added and modified.


LHCb partitioning Partition (TFC) Def. Generic term for a configurable

ensemble of parts of a sub-detector, an entire sub-detector or a combination of sub-detectors that can be run in parallel, independently and with a different timing, trigger and control configuration than any other partition.

Option: 16 or 32 concurrent partitions

Crucial: Equal internal propagation delays. If skew too large, FE will suffer from timing alignment problems when using different RS’.

INPUT CHANNELS

OUTPUT CHANNELS

Control Interface

Pool of Readout Supervisors

Partition A Partition B

Front-Ends grouped by TTCtx/Optical couplers to partition elements

TFC Switch

TTC

info

rmati

on e

nco

ded

ele

ctri

cal


Buffer overflows Throttle signals fed back electrically to the RS in control of the partition

with data congestion

Two Throttle Switches: Throttle signals to the L0 trigger Throttle signals to the L1 trigger

All Throttle Switches and ORs will log throttle history

INPUT CHANNELS

OUTPUT CHANNELS

OR OR

Control Interface

Pool of Readout Supervisors

Partition A Partition BThrottle Switch

Front-Ends etc grouped by Throttle ORs i.e. ~Throttle Switches

Thro

ttle

sig

nals


Progress and Status In view of the TDR, the aims of this year are to:

Design the TFC components specific to LHCb Review the TFC architecture and components Layout the first prototype of the TFC Switch and the RS Simulate the RS at several levels Test the way the TFC architecture exploits the TTC system Produce a first prototype of the TFC Switch and the RS Test critical points of the TFC Switch and the RS

Overview of work schedule for 2001:

Except for delays in the area of testing TTC system, schedule well maintained.

JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC

Testing TTC system

Testing TFC switch and TTC system

Testing RS


TFC Switch (Progress and Status)

Reviewed November 8, 2000 (together with TFC architecture and Throttle Switches)

very well received

First prototype was ready in April-May. Main aim with prototype was to measure the two crucial quantities:

Time skew between paths (aimed at <100 ps). Jitter (aimed at 50 ps)


TFC Switch (Progress and Status)

All measurements carried out successfully Jitter at the output ~80 - 100 ps. Jitter from generator ~50 ps Maximum skew between all inputs to each multiplexer is between 100 - 400 ps Skews between between output paths (multiplexers to output) was very large (-

> 4ns) A few mistakes were discovered in the routing when equalizing the paths.

• The mistakes + dielectric characteristics can account for the skews measured on the input and the ouput paths.

The intrinsic propagation delay of the multiplexers vary between 400 - 1000 ps. Specs claim maximum 850 ps.

Comparing line lengths with measured propagation delays shows that the signal speedis ~40% slower than the “ideal” 5ns/m. This is consistent in all measurements.

The measurements show that the performance with respect to skew is not satisfactory. Solution:

Route all lines on board layers with equal dielectric characteristics Add appropriate delay lines at the outputs to compensate for the inevitable

intrinsic skew due to the components.• Problem with temperature dependence of delay chips• Each board needs calibration.

Input and output coupling capacitors with less tolerance to improve signal shape.

Switch has still to be tested with CC-PC and in full TTC setup. The first prototype will be sufficient for tests of the first prototype of the RS


RS “Odin” (Progress and Status)

Specifications ready end of last year - almost entirely based on FPGA Specs, logical design and first draft of schematics reviewed April 4, 2001.

Very well received Importance of simulation emphasized.

Specs have been simulated in a high level behavioral model with a behavioral model of the LHC machine, trigger decision unit, and FE, using Visual HDL



The FPGA designs have been simulated using MaxPlus To check the FPGA designs and crosscheck the MaxPlus simulation, some

of the blocks have been simulated at gate level using Leapfrog.

The behavioral model of the LHC machine, the trigger decision units, and the FE have been refined in order to support a simulation of the real RS design. The behavioral model of the RS is currently being replaced in Visual HDL block by block by the FPGA designs at gate level including all delays.

The entire L0 path (except TTC encoder) has been simulated. Shows that the current design, using three or four clocks (different level of pipelining) works.

I/O

L0 pipeline

L0 handling



The interface to the L0 and the L1 trigger Decision Units have been agreed on.

RS Minimal Version currently in production: Almost all functionality but not the “RS internal FE” and fewer counters. Aim with first prototype is:

• Verify that the FPGAs are sufficiently fast with safe margin for the functions requiring synchronous operation.

• Measure performance and check concurrent operations of the many functions


TTC tests (Progress and Status)

The need for 1.1 MHz short broadcast transmission on channel B is a crucial point to test. Lacking RS, a test bench was setup using existing equipment:

Using a scope (before the TTCpr was available) shows no problem transmitting 1.1MHz short broadcasts. 1.6 MHz was measured. Data integrity not tested!

Since the encoder circuit in the TTCvx will implemented in the RS and we will use TTCtx the test bench has also allowed us to gain experience and study the performance.

TTCpr is designed to receive ATLAS L1A: Help from ATLAS to modify the code of the onboard FPGA to receive short

broadcasts. Two problems remain:

• The transfer of the short broadcasts into the host PC does not work properly.• Testing the same throughput (1.1 MHz * 16 bits) using the ATLAS version of

the FPGA (long broadcasts) shows problems above ~100 kHz. EventIDs show jumps. PC not capable to cope?

ALEPH FIC TTCvi TTCvx TTCtxTTCpr

VME


Conclusions LHCb TFC system architecture and specific components have been

reviewed in two reviews.

The partitioning concept well integrated.

The first designs and layouts of the LHCb specific components are ready.

Detailed simulation of RS continuing.

The first prototype of the TFC Switch built and the first RS in production.

The results of the tests of the first TFC Switch are very useful.

The first tests with the TTC system show positive results. Work going on with the TTCpr.

When the RS is ready, it will replace the TTCvi + TTCvx in the test bench

Date post:	14-Jan-2016
Category:	Documents
Upload:	melody
View:	25 times
Download:	1 times

LHCb Timing and Fast Control System

Documents