For more information on Pulsar board:http://hep.uchicago.edu/~thliu/projects/Pulsar/
Burkard Reisert (FNAL)
Nov. 7th, 2003
PULSAR Production Readiness
• Introduction
• Testing the PULSAR
• Tools involved
• People
Gigabit EthernetRF clock
SRAMs
Pulsar is designed to be: Modular, universal & flexible, fully self-testable (board &system level)
All interfaces are bi-directional (Tx & Rx) Lego-style, self test capability
Spare lines
one for all andall for one
user definedinterfaces
Personality cards
Standard link
another Pulsar or S-LINK to PCI
Has ALL interfaces L2 decision crate has
In God we trust … …the rest we test
Bottom view
Top view
Mezzanine slotsAUX card
S-LINK
General purpose, useful within & outside CDF
Pulsar
AUX Card
Pulsar
HotlinkTx/RxProd./testingdone
Taxi Tx/RxProd./testingdone
SLINK LSC/LDC (have some,need to order more)
ANL SLINK->GBE (production)
Pulsar Hardware
testing startedSix S32PCI64in hand
Ready for Production
Mezz. Cards
Mother Of All Boards
History
4 prototype Pulsar boards (late02/early03)
4 pre production Pulsar boards (August 03) (thin PCB 63mil instead of 96mil, vendor admitted error, boards still useful)
2 boards new pre production version (October 03)
Not a single blue wire No revisionPrototype Production
How We Tested
Tx Rx
1. Standalone Test Stand Mode
2. Test Stand: Pulsar PC
3. Pulsar in beam
self test all interfaces
PulsarA
UX
PC
• CERN Slink Cards• UofC Slink HOLA Card• ANL Slink to Giga Bit Ethernet
XTRP
RX
CDF MuonsTracks
CDFLevel2Trigger
4. Test procedure for Pulsars: See next slides
millions of events
millions of events
millions of events
millions of events
Registers
Registers
Registers
3 PowerLEDs
3.3, 2.5, 5V
Control FPGA
DataIO FPGA 1
VME chip
•Visual Inspection
•Put fuses, jumpers, Oscillators
• Power up
• JTAG-detect and program VME Chip, DataIO1&2, Control FPGA
•First VME Access to -- R only registers -- R/W register
• All Ok
DataIO FPGA 2
Initial Checkout of New Pulsar Boards
3 VMELEDs
JTAG
JTAG
JTAG
DataIO FPGA 1
VME chip
DataIO FPGA 2
VME access to SRAMs & internal RAMS
SRAM
SRAM
• Two 128Kx36 SRAMs on board
• Each DataIO FPGA has control of one SRAM
• Write and read SRAMs through VME
• Load SRAMs with 128K test patterns
• internal RAMs implemented in both DataIO+ControlFPGA
•Run Test Loop
• ALL OKControl FPGA
iRAM
iRAM
iRAM
Control FPGA
DataIO FPGA 1
VME chip
DataIO FPGA 2
P2 CDF control signals
• Pulsar FPGAs see ALL P2 CDF control signals
• Used Testclk to toggle all signals
• Signals recorded by circular buffer RAM in each FPGA
• Also checked with Logic Analyzer
• Data matchesRAM
RAM
RAM
P2
Control FPGA
DataIO FPGA 1
VME chip
DataIO FPGA 2
Pulsar P2 inter-communication lines
• Pulsar has five SVT style inter-communication lines on P2
• Data is sent from Control FPGA on one Pulsar and received by all three FPGAs on the other one
•Run test loop
• All Ok
P2
In/outregisters
Inputregister
Inputregister
Control FPGA
VME chip
Pulsar P3 spare lines
• Pulsar has 25 spare lines to P3
• Data sent from Control FPGA
• initial Check with Logic Analyzer from P3 back planelater with SVT input from AUX mapped to the spare lines
• All Ok Outputregister LALA
P3
Control FPGA
VME chip
TSI interface connection
TScable
• Data sent and received by Control FPGA
•Run test loop
• All Ok
In/outRegisters
SVT data path
SVT cableSVT data out
SVT data in
SVT data inSVT data in
Control FPGA
DataIO FPGA 1
DataIO FPGA 2
RAM
VME chip
• Input data is uploaded to Control FPGA RAM using VME
• Control FPGA sends SVT data out from SVT output
• All FPGAs receive incoming SVT data
• Data is read from each FPGA using VME
•Run test loop
• All Ok
Receiver FIFO
Receiver FIFO
Receiver FIFO
L1 data path: input and output
• Data is sent out from one Pulsar and received by another one
• Control FPGA sends data from internal RAM
• Data is received by all three FPGAs
• Read-out through VME
• Run test loop
• All Ok
DataIO FPGA 1
VME
DataIO FPGA 2
Control FPGA
Input register
VME
Control FPGAOutputRAM
Input register
Input register
Hotlink TX
Hotlink TX
Hotlink TX
Hotlink TX
Mezzanine Cards Interface
•Test Pattern loaded into Tx DataIO FPGA
• Data is driven out on L1As
•Data recorded intoRx DataIO FPGA Fifos
•Run test loop
•All Ok
• Tested with --Hotlink 20Mhz & 4x CDFClk-- TAXI 12MHz & CDFClk
RX
RX
RX
RX
16x
SLINK formatting
Control FPGA
DataIO FPGA 1
VME chip
DataIO FPGA 2
SpyFIFO
SpyFIFO
SpyFIFO
P3
•Load data into TX
•Receive data on Rx mezzanine cards
• DataIO FPGAs send merged input data in SLINK format to Control FPGA
• Control FPGA merges incoming data and sends it out in SLINK format from P3
• Outgoing data is stored into a Spy FIFO in each FPGA, and it can be read from the FIFOs using VME
• Data in the Control FPGA Spy FIFO matches the data uploaded to input Tx RAMs
LA
Data matches
SLINK Interface •Load Test Pattern / Event data into output RAM
• Outgoing SLINK data goes from P3 to AUX card, withLSC (link source card) LOOP BACK-- Receive data back on LDC (link destination card)
SLINK PCvia CernSLINK/HOLA/GBE-- Send event to PC -- Run L2 algorithms-- send decision backTiming measurement
• Run test loops
Pulsar AUX card
SL
INK
S
ourc
e C
ard
Des
tin
. Car
d
PC
Control FPGA
VME chip
Output RamReceiver Fifo
Loop back
ALL OK
Thanks to Kristian Hahn (Upenn)
Muon Input Data
XTRPinput Output
to TrkList
Pulsar as Level 2 Muon Interface Board
Alpha processor
12
matc
hbox &
4 p
rem
atc
h fi
bers
Pulsar receives-- XTRP data-- L2 Muon data
Pulsar appendsZero suppressedMuon data to Tracks
Pulsar sends Track & Muon dataTo L2 Alpha processor
(alpha code: unpacking of tracks & Muons L2 algorithm)
RunIIa L2 Muon path commission: methodology at work from discussion to error free path: < 3 months June Sept. 2003
L1 Muon (16 hotlink fibers)
andL1 track input
Pulsar muon+track data transmitter
upstream
Data source
Pulsar Muon Board LegacyL2 decision
crate
downstream
Data sink
Pulsar Receiver
We didn’t waste one second of beam time The FULL chain test with collision beam works on the first try (error free)
• Fully self-tested before put in the running exp. up to 1 Billions events in self-test mode.
Muon Input Data
XTRPinput Output
to TrkList
Pulsar as Level 2 Muon Interface Board
12
matc
hbox &
4 p
rem
atc
h fi
bers
What has been testedwith beam?
• Mezzanine Card Interface
• XTRP/SVT INPUT
• FPGA Algorithms at 80MHz
• Board internal data transfer
• XTRP/SVT OUTPUT
• CDF Control Signals
• DAQ readout
VME
CD
F S
ign
als
Test Runs SummarySTORE 2985 2003/09/03 21:49 2003/09/04 14:51 Run# L3A Lumi_live Max. rate Avg. rate Inst.Lumi (E30) (nb-1) L1 / L2 / L3 L1 / L2 / L3 start/end168766 335,930 196.220 15.0k/268/58 13.3k/255/53 35.0/30.2168767 155,974 85.949 13.7k/240/53 13.0k/230/50 29.5/28.0168775 596,981 258.197 12.5k/190/45 10.6k/169/39 19.7/15.8
STORE 2988 2003/09/04 21:26 2003/09/05 10:21Run# L3A Lumi_live Max. rate Avg. rate Inst.Lumi (E30) (nb-1) L1 / L2 / L3 L1 / L2 / L3 start/end168819 90,076 76.570 16.5k/148/30 13.8k/132/27 24.7/23.1168820 1,101,959 471.469 14.7k/235/50 12.0k/193/47 22.7/14.8168821 166,146 114.954 9.6k/ 94/22 8.6k/ 86/20 14.5/13.2168822 121,685 50.591 8.4k/134/34 7.3k/121/30 13.2/12.6
STORE 2988 2003/09/04 21:26 2003/09/05 10:21Run# L3A Lumi_live Max. rate Avg. rate Inst.Lumi (E30) (nb-1) L1 / L2 / L3 L1 / L2 / L3 start/end168889 2,816,155 1,359.049 15.5k/300/60 12.2k/205/45 40.0/13.7
ALL data collected for Pulsar beam test are good for physics analysis.No single Pulsar hardware/firmware problem was seen (~ 5M evts.)
PULSAR
TEST TOOLS
PULSAR PRODUCTION TEST SETUP
PC AU
X
Cra
te C
ontro
ller: T
STL2
TR
G1
TR
AC
ER
Pulsa
r Rece
iver (R
x)
Pulsa
r Tra
nsm
itter (T
x)
NEW BOARD
• Hardware andFirmware to test all pathsin place
• Automated software testing procedures in place
• Ready tofully testnew Pulsars
XTRP/SVT
Hotlink/Taxi
:: TSI
L1
Pulsar TEST VME Software
Test PatternGeneration &Analysis
TestClock UtilityFunctions
Pulsar Data Path Tests &
Utility Routines
Infrastructure:vxworks, VISION
Receive & Analyze Data on PC by Kristian
(Upenn)
Pu
lsar
Sta
nd
alo
ne T
est
s
Run Control Java GUI
FER-code
FrontEndReadout
Same package used for testing system firmware
Easy access throughJava GUI
(version by Mikko)
Automated Test Procedures
TestRAM() reading list of RAMs from steering file ram.test
TestSVT(1000000, 1, 1, 15, 15) SVT/XTRP input&output TestHotlink(12345, 1, 1, 7, 15) mezzanine card interfaceTestL1(111111111, 1, 1, 7, 15) L1Trigger bitsTestIntCom(98763, 1, 7, 15) Internal Communication lines :Pulsar_forever_andever(15) “infinite” SLINK loop back test
iterat
ions
pattern
type
L1A mode
Tx Slot
Rx Slot Tested interface
Too lazy to type? Use the PulsarTestGui !
Pulsar Test GUI0. select crate
2. Load code
1. reboot
3. Map crate
4. Select test, Tx Rx slot, start testDisplay results
for experts:•command line entry•read/write to reg.•resets
JavaGUI• Light weight• interfacing to Pulsar test c-routines
Man Power for Testing
People being trained for production testing:
• Vadim Rusu (New UC PostDoc) • Chris Neu (New Upenn PostDoc) • Mikko Hakala (FNAL, already trained)
Consultants:
• Burkard Reisert (FNAL)• Ted Liu (FNAL)
We have:• tested prototype & pre production boards -- all in test stand self test mode, all interfaces -- some as MUON/XTRP Rx in beam condition • well defined test procedures• necessary infrastructure in place -- Hardware (Crate, Tx/Rx Pulsar, Mezzanine cards) -- Software (VME c-code, JavaGUI)• well trained people
We are ready for production
Waiting for pulsar hurricane to come
Conclusion