CMS ESR December 2002 1

HTR Status and Clocking IssuesD. Baden, T. Grassi

http://www.physics.umd.edu/hep/esr_dec_2002.pdf

HCAL Trigger ReadoutHCAL Trigger ReadoutHCAL Trigger ReadoutHCAL Trigger Readout

CMS ESR December 2002 2

Shield Wall

SBS

HPD

FE MODULE

12 HTRs perReadout Crate,2 DCC

FRONT-ENDRBXReadout Box (On detector)

READ-OUT CrateTrigger Primitives

Fibers at 1.6 Gb/s3 QIE-channels per fiber

QIE

QIE

QIE

QIE

QIE

QIE

CC

A

GOL

DCC

TTC

GOL

CC

A

HTR

HTR

CAL

REGIONAL

TRIGGER

32 bits@ 40 MHz

16 bits@ 80 MHz

CC

A

S-Link: 64 bits @ 25 MHz

Rack CPU

FE/DAQ ElectronicsFE/DAQ ElectronicsFE/DAQ ElectronicsFE/DAQ Electronics

CLK

HTR

CMS ESR December 2002 3

HTR Principal FunctionsHTR Principal FunctionsHTR Principal FunctionsHTR Principal Functions

1. Receive front-end data for physics running

• Synchronize optical links

• Data validation and linearization

• Form TPG’s and transmit to Level 1 at 40 MHz

• Pipeline data, wait for Level 1 accept• Upon receiving L1A:

• Zero suppress, format, transmit to the concentrator (no filtering)

• Handle DAQ synchronization issues (if any)

2. Calibration processing and buffering of:

• Radioactive source calibration data

• Laser/LED calibration data

3. Support a VME data spy monitoring

CMS ESR December 2002 4

Readout VME CrateReadout VME CrateReadout VME CrateReadout VME Crate

“BIT3” board – Slow monitoring over VME– Commercial VME/PCI Interface to CPU

FanOut board– Takes TTC stream in– Clone and Fanout timing signals

HTR (HCAL Trigger and Readout) board– Spy output over VME– FE-Fiber input– TPG output (SLBs) to CRT– DAQ/TP Data output to DCC

DCC (Data Concentrator Card) board– Input from HTRs– Spy output– Output to DAQ

DCC

VME CRATE

20m Copper1.2 Gb/s

DAQ

Calorimeter Regional Trigger

BIT3

Fiber1.6 Gb/s

FanOut

HTR

Front End Electronics

HTR

DCC

HTR

HTR

...

TTC fiber

CMS ESR December 2002 5

• Board organized around 2 identical sets of circuitry:

• Optical inputs• 1.6 GHz, 8B/10B frames, 3ch/link• Dual LC detectors and drivers

• TI TLK2501 Deserializers• Crystal RefClk• TTC 80MHz backup

• Xilinx Virtex FPGA XCV1000E• 24 channels each

• TPG signals• Sent to SLB over backplane,

LVDS• SLBs mounted 6 to a transition

board• Level 1 accept output to DCC

• LVDS output

• VME• Altera FPGA and firmware

OLD DESIGN

““Old” HTR Design Old” HTR Design (Summer 2002)(Summer 2002)

““Old” HTR Design Old” HTR Design (Summer 2002)(Summer 2002)

CMS ESR December 2002 6

HTR Functional ExperienceHTR Functional ExperienceHTR Functional ExperienceHTR Functional Experience

• What was tested:• VME fully tested and working

• Some changes necessary to conform to CMS VME standards• Optical links and synchronization

• No indication of any problems. Big success here – was a real worry

• LVDS to DCC• Tested, working (Will change cable/connector to Cat 6/RJ45)

• Fanout of timing signals on two Cat5 cables• Plan to change to a single Cat6 or Cat7 cable (very low cross-talk)

• Firmware – full tests of:• Pipeline and L1A triggering capability• In-line histogramming for source calibration

• TTCrx• Not working at all (4 bad on 4 tested).

• What was not tested: Anything to do with TPG

CMS ESR December 2002 7

HTR Board ExperienceHTR Board ExperienceHTR Board ExperienceHTR Board Experience

• Produced ~12 boards• Several bare boards were delivered warped

• Many opens under FPGA after assembly (~9 boards)

• Some fixed after reflow (a few)

• Some worse after reflow (shorts)

• X-rayed a few boards, sometimes inconclusive

• Some opens on VME side

• Non BGA FPGA, indicates bad vias

• Few other various open circuits

• Finally got ~8 boards to “work”• Questionable reliability

CMS ESR December 2002 8

ModificationsModificationsModificationsModifications• Change board from using white-tin to gold traces

• This process was sold to us by the board maker. Our mistake.• Used only for very high volume, cost competitive products, very

difficult and expensive to control. • Gold is flatter and not very much more expensive (~$50/board),

better for FPGAs

• Change assembly house• Insufficient Quality Control on current assembler – they are fired.• We visited 2 high-end assemblers

• Modern Machines • Step up and step down oven temp control.• In-line X-ray for BGA QC• Manufacturability Review

• Add stiffeners to HTRS• Flexability of 9U VME boards was underestimated

• Worry: fine-line BGA (FBGA) can pop connections

CMS ESR December 2002 9

Modifications (cont)Modifications (cont)Modifications (cont)Modifications (cont)

• Change from FBGA to BGA• FBGA 1.0mm pitch, change to BGA 1.27mm pitch

• No additional expense, plenty of available real estate, no need to push

• We are just being very conservative here

• JTAG capabilities added• Will help with debugging

• By making these changes…• We have profited from the summer

• We have reduced our production risk considerably

CMS ESR December 2002 10

HTR Design ChangesHTR Design ChangesHTR Design ChangesHTR Design Changes• SLB transition board issues:

• Worries about so many LVDS signals over backplane for old design• Routing is too complicated

• Many signals going to same backplane location• Requires multi-layer routing with many vias

• TPG cables very thick• Mechanical issues are very worrisome

• SLB changes needed (e.g. height reduced after ECAL redesign…)• Solution: move SLB’s to HTR motherboard

• Benefits:• Mechanically attach SLB’s to HTR front panel for mechanical stability• Eases routing requirements, reduces board and assembly risks, cheaper too

• Change from Xilinx VirtexE to Virtex2• More resources, block ram, hardware multipliers• Big cost reduction (save $180k)• More modern chip for long-term maintenance

• Clock synchronization• Decouple “80MHz” crystal from FPGA system clock• Will allow us to use crystal to maintain synchronization of serdes

• This gives us 2 solutions for our “40ps” jitter requirement issue

CMS ESR December 2002 11

P1

to D

CC

New HTR Conceptual Design New HTR Conceptual Design New HTR Conceptual Design New HTR Conceptual Design

P2

LVDS

to L

evel

1 C

al T

rig

ger

LVDS

SLB

SLB

SLB

SLB

SLB

SLB

FPGAXilinxXC2V

LCLC

TITI

TI

TI

TI

TITI TI

LCLC

FPGAXilinxXC2V

LCLC

TITI

TI

TI

TI

TITI TI

LCLC

VMEFPGA

Fibers

No

P3!

8-way8-way

fro

m

Fan

ou

t

RJ45TTC

CMS ESR December 2002 12

Clocking SchematicClocking SchematicClocking SchematicClocking Schematic

TTCrxTTC

80 MHz LVPECL Crystal 1 to 8

Fanout

1 to 8 Fanout

Single width VME

BC0

80MHz

40MHz

SLB

SLB

SLB

SLB

SLB

SLB

TI(16)

FPGA

BC0BC0

40MHz 1 to 8 Fanout

80MHz

TTC mezzTTC TTC broadcast bus

Cat 6/7 quad cable (allows PECL)

TTC Fanout Board

QPLL

• Start with Fanout card• TTCrx Maryland mezzanine card or CERN TTCrm daughterboard• QPLL• Fanout on Cat6/7 quad twisted pair TTC, BC0, 40MHz, 80MHz

• In HTR:• Send TTC signal to TTCrx mezzanine board, access to all TTC signals• Send 80MHz clean clock (cleaned by QPLL) to mux

• Select 80MHz clean clock OR crystal to TI deserializers

80 MHz

40 MHz

CMS ESR December 2002 13

HCAL TRIDas Clock SchemeHCAL TRIDas Clock SchemeHCAL TRIDas Clock SchemeHCAL TRIDas Clock Scheme

TTCrx QPLL

(‘CC’ means Clean Clock)

Cat6/7 RJ45

RJ45 TTCMezz

TTC

SLB

Xilinx

TTC broadcast, L1A, BCR, EVR, CLK40

Fanout Card

4 twisted pair…

TTC BC0 CC40CC80

HTR Board

CC40

CC80

BC0

CMS ESR December 2002 14

Fanout – HTR schemeFanout – HTR schemeFanout – HTR schemeFanout – HTR scheme

HTR

TTC fiber

TTC LVDS

CLK803.3V-PECL

RX_BC0 LVDS

Cat6Eor Cat7cable

8 clks to TLKs

DS90LV001

Q1Q2Q3Q4Q5Q6Q7Q8

MC100LVE310 3.3V PECL

CLK403.3V-PECL

LVDSFanoutx 8

PCK953LVPECL-to-LVTTLFanout(top layer)

PCK953LVPECL-to-LVTTLFanout(top layer)

8 clks to TLKs + TPs

To 6SLBs

Diff. to 2Xilinx+ termin.

Diff. to 6 SLBsSingle-end to 2 xilinx

TTC daughter

card

ININ_b

Brdcst<7:2>,BrcstStr, L1A,BCntResto xilinxand SLBs

CLK80 LVDS

FanoutBoard

Low-jitterFanout x 15

O/E

Brdcst<7:2>,BrcstStr BC0

Fanoutbuffer

TTCTTC

TTC

FPGA

Fanout x 15

Brdcst<7:2>,BrcstStr,BCntRes,L1A

CMOSLVDSor diffPECL

……..……..

……..……..

15 connectorson bottomlayer ?

15 Cables &Connectorstbd

……..……..

NB100LVEP221is LVDS compatible

TTCrx

(or daughter card)

QPLL

AN1568/D Fig 11Onsemi.com

RJ45

~FifteenRJ45connectors

PECLfanout

e.g. DS90LV110

..

..

2 TestPoints forCLK40andBC0

CLK40 LVDS

PECLfanout

..

..

..

..

80.0789 MHz3.3V crystalDiff. PECL

MC100LVEL37

CKCKCK/2CK/2

..

..

..

..

9U Front-panel space = 325 mm ; => space per connector ~ 21.5 mm

Notes: SLBs require fanout of CLK40, BC0. FE-link possibly requires CLK80. PECL fanout was tested in TB2002. One Cat6E cable (low x-talk) replaces the 2 Cat5 cables used in TB2002. TTC and BC0 remain LVDS as in Weiming’s board.HTR needs Broadcast bus, BCntRes and L1A: from TTCrx if we get it to work, otherwise we have to fan them out.

LVDS

Tullio Grassi <tullio@physics.umd.edu>

CMS ESR December 2002 15

TTCrx Mezzanine cardTTCrx Mezzanine cardTTCrx Mezzanine cardTTCrx Mezzanine card

• Very simple card:• 2 PMC connectors

• TTCrx chip

• TTC signal driver on motherboard

• Will be sent out for prototype next week

• Used by HTR, DCC, and Fanout cards

CMS ESR December 2002 16

TTC Distribution – Fanout CardTTC Distribution – Fanout CardTTC Distribution – Fanout CardTTC Distribution – Fanout Card

• Currently HCAL has 6 TTC partitions:• Each partition requires TTCvi and TTCex

• Each HCAL VME crate will have a single TTCrx receiving data directly from TTCex in a single VME card (Fanout Card)

• Fanout TTC signal to HTR mezzanine card with TTCrx chip

• Use quad twisted pair CAT6/7 cable allows PECL fanout

• TTC raw, BC0, 40MHz clean, 80MHz clean fanout

• Cost savings and simplification

• TTC monitoring by Fanout card over VME

• Count resets, etc…

CMS ESR December 2002 17

Random Latency IssueRandom Latency IssueRandom Latency IssueRandom Latency Issue

•Texas Instruments TLK2501 Serdes• Run with 80MHz frame clock – 20 bits/frame, 1.6GHz bit clock

• 625ps bit time

• Latency from data sheet:

• ~20ns variation (overall latency between 47 and 67ns)

• Fiber to fiber alignment could cross a 40MHz bucket boundary.

•How to fix? • SLB “knows” this latency – we will read it out after each reset

• HCAL LED fast rise time

• Can pulse during abort gap and align channels

• Requires LED pulsing alignment

• FE will send BC0 signal on all fibers

•Will measure this alleged latency with new HTR boards

CMS ESR December 2002 18

TPG AlignmentTPG AlignmentTPG AlignmentTPG Alignment

• TPG alignment performed in SLB• Necessary: All HTRs will send common BC0 to SLB’s

within each of 16 VME crates

• Calibration procedure to be performed for crate-crate alignment

• Initial alignment with LEDs, laser, etc.

• Final alignment with LHC first beam data

• Use “1-bucket” running to check everything

• This will ensure successful alignment

CMS ESR December 2002 19

DAQ AlignmentDAQ AlignmentDAQ AlignmentDAQ Alignment

• DAQ data must also be aligned• Must know L1A bucket for zero suppression

• Only if we will do ZSP on 1 or 2 HCAL channels centered on L1A bucket

• If ZSP done with sum over 5 channels, then this alignment is not critical

• Solution: discussed in previous slide• Read from SLB• FE sending known ID after with fixed offset relative to

BC0 during abort gap• Comparison of the two for error checking

• DAQ check on BC0 in DCC for alignment• Will send BC0, BCN, and EVN with the data to DAQ

CMS ESR December 2002 20

MISC ErrorsMISC ErrorsMISC ErrorsMISC Errors

• What happens if DCC finds mismatch in EVN?• DCC will then issue resynch request to aTTS system

• Details not yet defined but is fully programmable

• Fiber Link/synchronization errors (GOL/TI)• Work out protocols to inform DCC

• Reset requests to aTTS as well

• FE Clock/GOL PLL link errors• If GOL loses synch, then transmitter will send out IDLE

characters

• IDLE characters are illegal in a pipelined system!

• HTR will trap on IDLE as a signal that FE/GOL is having trouble

CMS ESR December 2002 21

ScheduleScheduleScheduleSchedule

O N D J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N DFirmwareBoard layoutFab/assembly 20 boards will be built but not assem bled Pre-production HTR boardCheckoutBoard layout if needed

Fab/assembly if needed Production prototypeCheckoutProductionTestbeam ?Vertical Slice ?

2002 2003 2004 2005