project
the FC7 AMC for DAQ & control applications in CMS
Mark Pesaresi (Imperial College), Paschalis Vichoudis (CERN)
Magnus Hansen, Manoel Barros Marin, Francois Vasey (CERN) Greg Iles, Sarah Greenwood, Andrew Rose, Geoff Hall (Imperial College)
| FC7 | Mark Pesaresi | TWEPP 2014 2
content
introduction - concept - motivations - specifications
hardware overview
- highlights - high frequency performance
acceptance testing & manufacturing
- test stands - manufacturing & production experience
uses in CMS summary
| FC7 | Mark Pesaresi | TWEPP 2014 3
concept
the FC7 AMC is a flexible, uTCA compatible card for generic CMS data acquisition & system control uses
the FC7 is targeted for
o both production systems or prototyping / bench-top use
o uTCA for CMS applications (compatibility with AMC13) or standard uTCA
o users of high speed optical links up to 10Gbps
Imperial MP7
CERN GLIB
| FC7 | Mark Pesaresi | TWEPP 2014 4
design motivations (I)
basing on existing hardware helps to reduce development time to maturity
o especially with regards to reuse of firmware and software components
o fewer nasty surprises!
designed to be an evolution of
o the Gigabit Link Interface Board (GLIB) – a highly
successful project filling a similar role, with emphasis on GBT
link testing
using design components, novel features and layout guidelines from
o the MP7 – a high optical density O(Tbps) processing card to
be used in the upgraded CMS Calorimeter Trigger
| FC7 | Mark Pesaresi | TWEPP 2014 5
design motivations (II)
principally designed with two CMS users in mind
o the Trigger Control & Distribution System (TCDS) – rationalisation and upgrade of existing TTC
control system for installation in 2014/5; distribution of LHC clock, fast (e.g. trigger) and slow
commands, reception of synchronous detector status, centralised trigger rules, more partitions
o the Pixel Front End Driver (pixFED) – replacement front end board to acquire data from the
upgrade pixel detector, to be installed in 2017; 400Mbps digital optics from front ends, increased
readout bandwidth to DAQ, robustness to readout errors, increased processing capability
| FC7 | Mark Pesaresi | TWEPP 2014 6
design motivations (II)
principally designed with two CMS users in mind
o the Trigger Control & Distribution System (TCDS) – rationalisation and upgrade of existing TTC
control system for installation in 2014/5; distribution of LHC clock, fast (e.g. trigger) and slow
commands, reception of synchronous detector status, centralised trigger rules, more partitions
o the Pixel Front End Driver (pixFED) – replacement front end board to acquire data from the
upgrade pixel detector, to be installed in 2017; 400Mbps digital optics from front ends, increased
readout bandwidth to DAQ, robustness to readout errors, increased processing capability
- compatibility with uTCA for CMS (AMC13)
- low jitter clock distribution & deterministic latency
- capability to run 10Gbps SERDES links
| FC7 | Mark Pesaresi | TWEPP 2014 7
design motivations (II)
principally designed with two CMS users in mind
o the Trigger Control & Distribution System (TCDS) – rationalisation and upgrade of existing TTC
control system for installation in 2014/5; distribution of LHC clock, fast (e.g. trigger) and slow
commands, reception of synchronous detector status, centralised trigger rules, more partitions
o the Pixel Front End Driver (pixFED) – replacement front end board to acquire data from the
upgrade pixel detector, to be installed in 2017; 400Mbps digital optics from front ends, increased
readout bandwidth to DAQ, robustness to readout errors, increased processing capability
- compatibility with uTCA for CMS (AMC13)
- low jitter clock distribution & deterministic latency
- capability to run 10Gbps SERDES links
additionally: continued capability to run GBT (4.8Gbps) systems
| FC7 | Mark Pesaresi | TWEPP 2014 8
FC7: top view
looks like a
simple board…
what’s the
fuss?
| FC7 | Mark Pesaresi | TWEPP 2014 9
FC7: bottom view
OK
that might
explain why
it’s so heavy…
| FC7 | Mark Pesaresi | TWEPP 2014 10
main features
- full size, double width AMC module for uTCA
- Kintex 7 FPGA supporting serial links up to 10Gbps
- 4Gb DDR3 RAM (128M x 32b)
- 2 FMC sockets fully compatible with Low Pin Count FMCs
o additionally supports up to 20 high speed serial
lanes to the front panel
o supports dual-width, legacy & non-standard FMCs
- flexible backplane configuration
o high speed connectivity on up to 12 ports
o fully compatible with uTCA for CMS and AMC13
dual QSFP+
8 x SFP+
2x 14b A/D, 2x 16b D/A
| FC7 | Mark Pesaresi | TWEPP 2014 11
development history
November 2012
June 2013
July /August 2013
September 2013
November 2013
January 2014
June/July 2014
Today
Spec approved
First prototypes (R0a) submitted
First prototypes received Majority of design validated, one minor design issue Some serious manufacturing issues uncovered
Begin assembly of test system & firmware Informal price enquiry to trial new manufacturers
Pre-production series (R0b) submitted with two manufacturers
Pre-production batches received Excellent manufacturing quality with one supplier Order for 70 pieces placed
70 production boards received (TCDS)
Minor design update (R1) submitted Order of 120 pieces in production (pixFED)
dev
elo
pm
ent
pro
du
ctio
n
| FC7 | Mark Pesaresi | TWEPP 2014 12
feature highlights
Kintex 7 FPGA
high range Kintex 7 XC7K420T
pin compatible with XC7K480T (difference only in logic)
FMC 1
FC7 is FMC spec compliant but can also accommodate non-standard mezzanines &
stacking heights
‘-2’ speed grade – MGTs support 10Gbps (10.3125Gbps max)
supports almost any MGT protocol to FMCs (e.g. GBT, 10GbE, custom)
12 MGT pairs
68 user IO (34 differential)
I/O operates up to 3.3V signalling - supports legacy FMCs
| FC7 | Mark Pesaresi | TWEPP 2014 13
feature highlights
switched JTAG header FPGA, microcontroller, FMCs
backplane JTAG option
unique laser etched QR code unique EEPROM h/w address
automatic logging & inventory,
configuration with CERN assigned MAC address on acceptance
external 12V power header
desktop/crate mode switch (requires additional h/w for GbE control)
| FC7 | Mark Pesaresi | TWEPP 2014 14
feature highlights
DDR3SRAM
AM
C c
on
ne
cto
r
4Gb DDR3 specified up to 480MHz
30Gbps maximum bandwidth (R+W)
designed to be compatible with both uTCA for CMS and
other AMC.x standards
supports range of MGT protocols across backplane
Port 0 GbE
Port 1 GbE or AMC13-DAQ*
Port 2 SATA/SAS
Port 3 SATA/SAS or AMC13-TTC/S (LVDS)
Ports 4-7 PCIe/SRIO*
Ports 8-11 Ext Fat-Pipe (SRIO/10GbE XAUI)*
Ports 12-15 LVDS I/O
Ports 17-20 unused
FCLKA AMC13-TTC or PCIe clocks
TCLKA/TCLKC telecoms clock inputs
TCLKB/TCLKD telecoms clock outputs
| FC7 | Mark Pesaresi | TWEPP 2014 15
feature highlights
flash
uC
uSD
firmware image repository including a fallback inviolate ‘Golden’ image
allows fast write (~10s) speeds & instant
FPGA reconfiguration over IPBus (compared to 5min for JTAG->SPI flash)
ATMEL microcontroller runs imperial_mmc v1.6 code
(shared with MP7)
| FC7 | Mark Pesaresi | TWEPP 2014 16
feature highlights
16 layer PCB Nelco N4000 13-EPSI - low loss tangent for HF - low dielectric constant -misaligned with PCB weave
strict constraint on thickness
| FC7 | Mark Pesaresi | TWEPP 2014 17
prototype performance testing
electrical TX testing at 10Gbps
XILINX SERDES electrical breakout FMC (XM104)
optical TX & loopback testing at 10Gbps
Faster Technology 8x SFP+ FMC (FM-S18)
20ps total jitter
1.7ps ch-ch spread
specification <27ps
BER <10-12 (PRBS31)
performance limited by
optical margin/SFPs
| FC7 | Mark Pesaresi | TWEPP 2014 18
production acceptance testing (I)
GPIB-controllable power supply
GPIB-USB controller
ATMEL programmer (microcontroller configuration)
XILINX programmer (CPLD configuration)
SD card (FPGA configuration)
| FC7 | Mark Pesaresi | TWEPP 2014 19
production acceptance testing (II)
AMC extender card (CERN) Ports 1-15 loopback Port 0 IPBus control
electrical loopback FMCs (commercial)
| FC7 | Mark Pesaresi | TWEPP 2014 20
production acceptance testing (III)
- visual inspection
- scan QR code
- connect to the testbench
1. Power up 2. Configure CPLD 3. Configure microcontroller 4. Power cycle to load FPGA (from SD) 5. Check ethernet link with FPGA (ping) 6. Verify on-board voltages (via the monitoring circuitry) 7. Identify the board (i2c readable S/N) 8. Assign MAC address (from LUT based on label) 9. Run DDR3 memory test (@960MT/s) 10. Run FMC loopback test (standard I/O) 11. Increase current limit for SERDES tests 12. Run FMC loopback test (SERDES @ 10Gb/s) 13. Run AMC loopback test (SERDES @ 5Gb/s) 14. Power down
| FC7 | Mark Pesaresi | TWEPP 2014 21
production acceptance testing (III)
- visual inspection
- scan QR code
- connect to the testbench
1. Power up 2. Configure CPLD 3. Configure microcontroller 4. Power cycle to load FPGA (from SD) 5. Check ethernet link with FPGA (ping) 6. Verify on-board voltages (via the monitoring circuitry) 7. Identify the board (i2c readable S/N) 8. Assign MAC address (from LUT based on label) 9. Run DDR3 memory test (@960MT/s) 10. Run FMC loopback test (standard I/O) 11. Increase current limit for SERDES tests 12. Run FMC loopback test (SERDES @ 10Gb/s) 13. Run AMC loopback test (SERDES @ 5Gb/s) 14. Power down
| FC7 | Mark Pesaresi | TWEPP 2014 22
manufacturing experiences
submitted prototype and pre-production series runs with two different manufacturers
o one well known to CERN/UK, one with no prior experience of
o experimented with two PCB materials compatible with the impedances & high speed
signal integrity required
large fraction of development period dedicated to tracking down & resolving manufacturing issues
o two manufacturers extremely useful to identify manufacturing vs. design faults (whether
due to designer error or board complexity)
o have learnt the importance of good, two-way communication with your supplier
| FC7 | Mark Pesaresi | TWEPP 2014 23
example 1: PCB issues
for 10Gbps we can’t use simple FR4…
Nelco N4000 laminate (Park) has very good high frequency characteristics
but now known to be susceptible to delamination & registration defects
o also an expensive material, low yield
o requires careful handling during manufacture & assembly;
e.g. contamination, humidity, life time, temperature profiles
o working guidelines between PCB/assembly manufacturers compiled
TU-872 SLK laminate (TUC) trialled successfully but also tends to suffer
similar registration defects – special measures during construction
even more fancy HF materials now available but even less well known
o key is building communicative relationship with supplier real x-ray representations
| FC7 | Mark Pesaresi | TWEPP 2014 24
example 2: assembly issues
concerning assembly, one of the suppliers demonstrated an excellent
quality of work
continuous improvement in the assembly process
o detailed written feedback provided after each batch
o use X-ray imaging to identify soldering issues
o identified washing process behind an issue with a mechanical
switch – assembly process changed accordingly
o improved packaging guidelines after damage during transport
| FC7 | Mark Pesaresi | TWEPP 2014 25
pre & production yield
best manufacturer chosen for supply of production FC7s
total 80 pre-production and production boards manufactured between Nov ’13 - Jun ‘14
70 passed all tests first time around, 8 returned to manufacturer
78 passed tests second time around
o 1 board damaged in transit
o 1 board with unidentified issue on backplane
final yield >97%
| FC7 | Mark Pesaresi | TWEPP 2014 26
firmware
mature firmware
development
- based on IPBus
system related logic is
segregated from user
reuse of GLIB f/w blocks,
similar user logic structure
suite of scripts for
configuration over IPBus
high level f/w diagram
| FC7 | Mark Pesaresi | TWEPP 2014 27
FC7 for CMS TCDS
with 3 custom FMCs, the FC7 satisfies the role of 2 of 3 objects for the TCDS
a local partition manager <-- interfaces with AMC13, DAQ, ext. clk/trig, partition interface cards
a partition interface <-- interfaces with partition managers, new (uTCA) & legacy FEDs
AMC13
local partition managers
partition interface cards
P5
| FC7 | Mark Pesaresi | TWEPP 2014 28
FC7 for CMS TCDS
with 3 custom FMCs, the FC7 satisfies the role of 2 of 3 objects for the TCDS
a local partition manager <-- interfaces with AMC13, DAQ, ext. clk/trig, partition interface cards
a partition interface <-- interfaces with partition managers, new (uTCA) & legacy FEDs
AMC13
local partition managers
partition interface cards
TTC/TTS to-from AMC13
to DAQ
TTC/TTS
TTC/TTS to-from FEDs
P5
| FC7 | Mark Pesaresi | TWEPP 2014 29
FC7 for CMS pixFED
with a single custom FMC, the FC7 can fully satisfy the role of the pixel FED
2x 12 channel parallel pluggable optical receivers (Fitel)
- running at 400Mbps, reads in data directly from detector
SFP+/QSFP+ required for DAQ (not pictured)
- 10Gbps readout, to be prototyped
total 70 boards required for pixFED, possibly more
with the TCDS FMC, the FC7 could also satisfy the role
of the pixel Front End Controller (FEC)
nothing special required, R&D ongoing
~30 boards required for pixFEC
| FC7 | Mark Pesaresi | TWEPP 2014 30
summary
flexible Kintex 7 based uTCA compatible AMC developed for generic DAQ/detector control uses in CMS
o suitable both for small scale test systems or production systems , compatible with a range of FMCs
o able to operate up to 10Gbps line rates across the front panel
o advanced FPGA configuration features
successful production of 80 pieces to date, another batch of 120 currently in production
o short development timescale
o mature system firmware, isolated from user firmware, test script suite
stress the importance of good communication with your supplier
o especially helpful when requesting non standard PCB constructions
o automated acceptance testing for quickly identifying manufacture issues
FC7s already being installed in P5
o TCDS ready for operation in 2015
o pixel FED development under way
| FC7 | Mark Pesaresi | TWEPP 2014 31
links
https://espace.cern.ch/project-FC7
firmware & software:
https://svnweb.cern.ch/cern/wsvn/ph-ese/be/fc7/
public documentation:
https://svnweb.cern.ch/cern/wsvn/ph-ese/be/fc7/trunk/fc7_r0/doc/
project
thank you for your attention
with special thanks to: Jan Troska, Jeroen Hegeman, Karl Gill