For more information on Pulsar board:http://hep.uchicago.edu/~thliu/projects/Pulsar/

Burkard Reisert (FNAL)

Nov. 7th, 2003

PULSAR Production Readiness

• Introduction

• Testing the PULSAR

• Tools involved

• People

Gigabit EthernetRF clock

SRAMs

Pulsar is designed to be: Modular, universal & flexible, fully self-testable (board &system level)

All interfaces are bi-directional (Tx & Rx) Lego-style, self test capability

Spare lines

one for all andall for one

user definedinterfaces

Personality cards

Standard link

another Pulsar or S-LINK to PCI

Has ALL interfaces L2 decision crate has

In God we trust … …the rest we test

Bottom view

Top view

Mezzanine slotsAUX card

S-LINK

General purpose, useful within & outside CDF

Pulsar

AUX Card

Pulsar

HotlinkTx/RxProd./testingdone

Taxi Tx/RxProd./testingdone

SLINK LSC/LDC (have some,need to order more)

ANL SLINK->GBE (production)

Pulsar Hardware

testing startedSix S32PCI64in hand

Ready for Production

Mezz. Cards

Mother Of All Boards

History

4 prototype Pulsar boards (late02/early03)

4 pre production Pulsar boards (August 03) (thin PCB 63mil instead of 96mil, vendor admitted error, boards still useful)

2 boards new pre production version (October 03)

Not a single blue wire No revisionPrototype Production

How We Tested

Tx Rx

1. Standalone Test Stand Mode

2. Test Stand: Pulsar PC

3. Pulsar in beam

self test all interfaces

PulsarA

UX

PC

• CERN Slink Cards• UofC Slink HOLA Card• ANL Slink to Giga Bit Ethernet

XTRP

RX

CDF MuonsTracks

CDFLevel2Trigger

4. Test procedure for Pulsars: See next slides

millions of events

millions of events

millions of events

millions of events

Registers

Registers

Registers

3 PowerLEDs

3.3, 2.5, 5V

Control FPGA

DataIO FPGA 1

VME chip

•Visual Inspection

•Put fuses, jumpers, Oscillators

• Power up

• JTAG-detect and program VME Chip, DataIO1&2, Control FPGA

•First VME Access to -- R only registers -- R/W register

• All Ok

DataIO FPGA 2

Initial Checkout of New Pulsar Boards

3 VMELEDs

JTAG

JTAG

JTAG

DataIO FPGA 1

VME chip

DataIO FPGA 2

VME access to SRAMs & internal RAMS

SRAM

SRAM

• Two 128Kx36 SRAMs on board

• Each DataIO FPGA has control of one SRAM

• Write and read SRAMs through VME

• Load SRAMs with 128K test patterns

• internal RAMs implemented in both DataIO+ControlFPGA

•Run Test Loop

• ALL OKControl FPGA

iRAM

iRAM

iRAM

Control FPGA

DataIO FPGA 1

VME chip

DataIO FPGA 2

P2 CDF control signals

• Pulsar FPGAs see ALL P2 CDF control signals

• Used Testclk to toggle all signals

• Signals recorded by circular buffer RAM in each FPGA

• Also checked with Logic Analyzer

• Data matchesRAM

RAM

RAM

P2

Control FPGA

DataIO FPGA 1

VME chip

DataIO FPGA 2

Pulsar P2 inter-communication lines

• Pulsar has five SVT style inter-communication lines on P2

• Data is sent from Control FPGA on one Pulsar and received by all three FPGAs on the other one

•Run test loop

• All Ok

P2

In/outregisters

Inputregister

Inputregister

Control FPGA

VME chip

Pulsar P3 spare lines

• Pulsar has 25 spare lines to P3

• Data sent from Control FPGA

• initial Check with Logic Analyzer from P3 back planelater with SVT input from AUX mapped to the spare lines

• All Ok Outputregister LALA

P3

Control FPGA

VME chip

TSI interface connection

TScable

• Data sent and received by Control FPGA

•Run test loop

• All Ok

In/outRegisters

SVT data path

SVT cableSVT data out

SVT data in

SVT data inSVT data in

Control FPGA

DataIO FPGA 1

DataIO FPGA 2

RAM

VME chip

• Input data is uploaded to Control FPGA RAM using VME

• Control FPGA sends SVT data out from SVT output

• All FPGAs receive incoming SVT data

• Data is read from each FPGA using VME

•Run test loop

• All Ok

Receiver FIFO

Receiver FIFO

Receiver FIFO

L1 data path: input and output

• Data is sent out from one Pulsar and received by another one

• Control FPGA sends data from internal RAM

• Data is received by all three FPGAs

• Read-out through VME

• Run test loop

• All Ok

DataIO FPGA 1

VME

DataIO FPGA 2

Control FPGA

Input register

VME

Control FPGAOutputRAM

Input register

Input register

Hotlink TX

Hotlink TX

Hotlink TX

Hotlink TX

Mezzanine Cards Interface

•Test Pattern loaded into Tx DataIO FPGA

• Data is driven out on L1As

•Data recorded intoRx DataIO FPGA Fifos

•Run test loop

•All Ok

• Tested with --Hotlink 20Mhz & 4x CDFClk-- TAXI 12MHz & CDFClk

RX

RX

RX

RX

16x

SLINK formatting

Control FPGA

DataIO FPGA 1

VME chip

DataIO FPGA 2

SpyFIFO

SpyFIFO

SpyFIFO

P3

•Load data into TX

•Receive data on Rx mezzanine cards

• DataIO FPGAs send merged input data in SLINK format to Control FPGA

• Control FPGA merges incoming data and sends it out in SLINK format from P3

• Outgoing data is stored into a Spy FIFO in each FPGA, and it can be read from the FIFOs using VME

• Data in the Control FPGA Spy FIFO matches the data uploaded to input Tx RAMs

LA

Data matches

SLINK Interface •Load Test Pattern / Event data into output RAM

• Outgoing SLINK data goes from P3 to AUX card, withLSC (link source card) LOOP BACK-- Receive data back on LDC (link destination card)

SLINK PCvia CernSLINK/HOLA/GBE-- Send event to PC -- Run L2 algorithms-- send decision backTiming measurement

• Run test loops

Pulsar AUX card

SL

INK

S

ourc

e C

ard

Des

tin

. Car

d

PC

Control FPGA

VME chip

Output RamReceiver Fifo

Loop back

ALL OK

Thanks to Kristian Hahn (Upenn)

Muon Input Data

XTRPinput Output

to TrkList

Pulsar as Level 2 Muon Interface Board

Alpha processor

12

matc

hbox &

4 p

rem

atc

h fi

bers

Pulsar receives-- XTRP data-- L2 Muon data

Pulsar appendsZero suppressedMuon data to Tracks

Pulsar sends Track & Muon dataTo L2 Alpha processor

(alpha code: unpacking of tracks & Muons L2 algorithm)

RunIIa L2 Muon path commission: methodology at work from discussion to error free path: < 3 months June Sept. 2003

L1 Muon (16 hotlink fibers)

andL1 track input

Pulsar muon+track data transmitter

upstream

Data source

Pulsar Muon Board LegacyL2 decision

crate

downstream

Data sink

Pulsar Receiver

We didn’t waste one second of beam time The FULL chain test with collision beam works on the first try (error free)

• Fully self-tested before put in the running exp. up to 1 Billions events in self-test mode.

Muon Input Data

XTRPinput Output

to TrkList

Pulsar as Level 2 Muon Interface Board

12

matc

hbox &

4 p

rem

atc

h fi

bers

What has been testedwith beam?

• Mezzanine Card Interface

• XTRP/SVT INPUT

• FPGA Algorithms at 80MHz

• Board internal data transfer

• XTRP/SVT OUTPUT

• CDF Control Signals

• DAQ readout

VME

CD

F S

ign

als

Test Runs SummarySTORE 2985        2003/09/03 21:49  2003/09/04 14:51 Run#         L3A  Lumi_live  Max. rate     Avg. rate     Inst.Lumi (E30)                      (nb-1)  L1 / L2 / L3   L1 / L2 / L3   start/end168766    335,930    196.220  15.0k/268/58   13.3k/255/53   35.0/30.2168767    155,974    85.949  13.7k/240/53   13.0k/230/50  29.5/28.0168775    596,981   258.197  12.5k/190/45   10.6k/169/39  19.7/15.8

STORE 2988        2003/09/04 21:26  2003/09/05 10:21Run#         L3A  Lumi_live  Max. rate       Avg. rate       Inst.Lumi (E30)                      (nb-1)  L1 / L2 / L3   L1 / L2 / L3   start/end168819     90,076    76.570  16.5k/148/30    13.8k/132/27   24.7/23.1168820  1,101,959   471.469  14.7k/235/50   12.0k/193/47  22.7/14.8168821    166,146   114.954   9.6k/ 94/22    8.6k/ 86/20  14.5/13.2168822    121,685    50.591   8.4k/134/34    7.3k/121/30  13.2/12.6

STORE 2988        2003/09/04 21:26  2003/09/05 10:21Run#         L3A  Lumi_live  Max. rate       Avg. rate       Inst.Lumi (E30)                      (nb-1)  L1 / L2 / L3   L1 / L2 / L3   start/end168889    2,816,155    1,359.049  15.5k/300/60    12.2k/205/45   40.0/13.7

ALL data collected for Pulsar beam test are good for physics analysis.No single Pulsar hardware/firmware problem was seen (~ 5M evts.)

PULSAR

TEST TOOLS

PULSAR PRODUCTION TEST SETUP

PC AU

X

Cra

te C

ontro

ller: T

STL2

TR

G1

TR

AC

ER

Pulsa

r Rece

iver (R

x)

Pulsa

r Tra

nsm

itter (T

x)

NEW BOARD

• Hardware andFirmware to test all pathsin place

• Automated software testing procedures in place

• Ready tofully testnew Pulsars

XTRP/SVT

Hotlink/Taxi

:: TSI

L1

Pulsar TEST VME Software

Test PatternGeneration &Analysis

TestClock UtilityFunctions

Pulsar Data Path Tests &

Utility Routines

Infrastructure:vxworks, VISION

Receive & Analyze Data on PC by Kristian

(Upenn)

Pu

lsar

Sta

nd

alo

ne T

est

s

Run Control Java GUI

FER-code

FrontEndReadout

Same package used for testing system firmware

Easy access throughJava GUI

(version by Mikko)

Automated Test Procedures

TestRAM() reading list of RAMs from steering file ram.test

TestSVT(1000000, 1, 1, 15, 15) SVT/XTRP input&output TestHotlink(12345, 1, 1, 7, 15) mezzanine card interfaceTestL1(111111111, 1, 1, 7, 15) L1Trigger bitsTestIntCom(98763, 1, 7, 15) Internal Communication lines :Pulsar_forever_andever(15) “infinite” SLINK loop back test

iterat

ions

pattern

type

L1A mode

Tx Slot

Rx Slot Tested interface

Too lazy to type? Use the PulsarTestGui !

Pulsar Test GUI0. select crate

2. Load code

1. reboot

3. Map crate

4. Select test, Tx Rx slot, start testDisplay results

for experts:•command line entry•read/write to reg.•resets

JavaGUI• Light weight• interfacing to Pulsar test c-routines

Man Power for Testing

People being trained for production testing:

• Vadim Rusu (New UC PostDoc) • Chris Neu (New Upenn PostDoc) • Mikko Hakala (FNAL, already trained)

Consultants:

• Burkard Reisert (FNAL)• Ted Liu (FNAL)

We have:• tested prototype & pre production boards -- all in test stand self test mode, all interfaces -- some as MUON/XTRP Rx in beam condition • well defined test procedures• necessary infrastructure in place -- Hardware (Crate, Tx/Rx Pulsar, Mezzanine cards) -- Software (VME c-code, JavaGUI)• well trained people

We are ready for production

Waiting for pulsar hurricane to come

Conclusion