24
FELIX Phase-II PCIe Card Versal Prototype Design Hongbin Liu ([email protected])
H.Liu
Versal VM1802 Introduction Part Number: XCVM1802-1MSEVSVA2197- ES9780 GTY Banks
11 Banks: bank 200 ~ 206 , bank 103 ~ 106 44 x GTY: up to 26.5625 for -1M(0.80V) devices 4 x PCIe Gen4 x 8 end-points
IO Banks 12 x XPIO banks: bank 700 ~ 711
VCCO: 1.0 V ~ 1.5V Group into four triplets (four DDR controller) 54 pins / 27 differential pairs per bank 9 x clock capable input pair per bank
2 x HDIO banks: bank 306 and 406 VCCO: 1.8 ~ 3.3 V 22 pins / 11 differential pairs per bank 2 x clock capable pins per bank
PMC (Platform Management Controller) 2 x PMC banks: bank 500 and 501
VCCO: 1.8 ~ 3.3 V 26 PMC MIO pins per bank (PMC or LPD accessible)
2
Processing System FPD (Full Power Domain, 2 x A72) LPD (Low Power Domain, 2 x R5F) 1 x LPD bank: bank 502
VCCO: 1.8 ~ 3.3 V 26 x LPD MIO pins per bank
H.Liu
Estimated by Xilinx Versal Prime XPE FPGA Utilization Configuration
Ambient Temp: 25 °C Effective ΘJA: 0.9 °C/W
Logic 70% Registers 70% LUTs (320 MHz, 12.5% toggle rate)
60% as logic 5% as shift registers and distributed RAMs
GTY Configuration (44 x 25 Gbps) 16 x 16 Gbps PCIe Gen4 24 x 25 Gbps FE links 4 x 16 Gbps PCIe/CCIX Gen4 links for SSD
With 25 Gbps increase total power by 0.3W Internal RAM (320 MHz clock, 25% enable rate)
90% BRAM Utilization 90% URAM Utilization
3 x 64-bit DDR4 controller @ 3200 Mbps data rate PS Configuration
2 x R5 core @ 600 MHz with 100% load 2 x A72 core @ 1.3 GHz with 100% load Interconnects @ 667 MHz with 100% load
75% DSP58 utilization, 25% toggle rate
Power - Power Estimation
3
H.Liu
Block Diagram of the Card
ADI uModule DC/DC power solution 1 x LTM4700 for VCCINT (up to 100A) 4 x LTM4678 for other power rails PMBus integrated for all DC/DC modules INA226 as back-up power monitoring ADM1066 for power sequence/marginal control
PCIe Gen4 x 16, up to 256 GT/s 16 x GTY links (16 Gbps, CCIX not supported) PCIe Gen4 switch for dual x8 to x16 conversion
12 x FE-Links One pair of Samtec Firefly 25G 12-Ch module
3 x Mini-UDIMM DDR4 Modules w/ ECC Accessible by both PL and PS through NoC
FMC + connector for mezzanine card ATLAS/DUNE variant 34 x differential pairs from XPIO banks 16 x GTY links FMC JTAG implemented 12V, 3.3V, 1.5V (VADJ) connected and monitored
4
Features
JTAG Header connector: Xilinx platform cable Digilent USB-JTAG module integrated
Booting: micro SD 3.0 and QSPI flash USB - I2C/UART GbE for debugging GPIO switches and LEDs
H.Liu
Power - Power Estimation
5
VM1802 Power Rail Sharing and sequence
Total worst case power: ~135W (With Efficiency: 0.85 ~ 0.90)
FPGA: 82W Other components: 56W
3 x DDR4: 20 W 1 pair of FF: 11W PEX: 22 W
Power limit of the PCIe Gen3 is 300W Auxiliary 8-pin power connector: 225W PCIe slot: 75W
Other components on board
H.Liu
Power - Distribution
6
Power solution VCCINT: LTM4700 (dual 50 A) 3 x LTM4678 (dual 25 A) 2 x LTM4642 (dual 4 A)
Power-rail voltage monitoring and sequence are done by two ADM1066
H.Liu
Power - Sensor, Monitoring, PG LED Current Sensor (Power Jumper)
Implemented on 10 power rails (except MGTYVCCAUX and DDR_VTT) VCCINT: Stackpole CSS2725FTL250
250 µΩ, 1%, 4 W, 2725, 6.81 mm x 6.45 mm P @ 100A = 2.5 W
Other 9 rails: Vashay WSK0612, 1mΩ, 3mm x 1.5 mm, 1 W P @ 25A = 0.625W
Power Monitoring 10 board power rails
Integrated PMBus on DC/DC regulators INA226
16-bit, shunt voltage LSB is 2.5 uV Alert to I2C - GPIO
3 rails to FMC+ 12V, 3.3V, VADJ (1.2V) INA226 monitoring through WSK0612 1mΩ current sensor
Power Good LEDs Implemented for 10 power rails Controlled by the PG signal of the DC/DC modules
7
H.Liu
Thermal Solution
8
Circuitry reference to VMK180 Sch. PWM fan controller
MAX6643 Automatically adjusted according to FPGA temperature (SYSMON_DXP/N)
VM1802 (Optional) PWM and FAN_TACH connected to HDIO bank 406
Signals to VM1802 HDIO GPIO: FANFAILT, OT_B, FULLSP Customized FPGA heatsink from Alpha Nova Tech. purchased
54 x 54 mm, 20/15 mm height, M45 fan, TIM: TPCM 585
H.Liu
GTY Quad Ref. CLK Accessibility
Clock Distribution VM1802 System Clocks
Bank 706: 200M fixed Bank 707: 100M adj. Bank 708: 2 x SMP, PCIe Buf CLK
Ref. clocks to/from FMC+ 2 to FMC+: 1 from each Si5345 3 from FMC+: GBTCLK[0:2]_M2C
2 x Si5345 PLL/Jitter cleaner 0-delay mode implemented (not
shown on diagram, OUT9 => IN3 CTRL signals by VM1802 HDIO SI5345 #A
Input: XPIO bank 710, Si570 (40.08M), SMP, OUT9
SI5345 #B Input: SI5345#A.OUT8, Si570
(40.08M), XPIO bank 711, OUT9
9
FE-Links Outputs from two SI5345
PCIe Links Outputs from two SI5345 PCIe Ref. CLK from the motherboard
FMC Links Outputs from two SI5345 GBTCLK from FMC+
H.Liu
VM1802 - JTAG Configuration Two JTAG paths implemented
1.8V voltage level JTAG header Digilent USB-JTAG module (USB to front-panel)
FMC+ JTAG connected 1.8V to 3.3V level translator Use FMC PRSNT to control the shunt of FMC TDI to TDO
Two resistor shunt path for TDI to TDO
10
87832-1420(On board)
Digilent JTAG-HS3 cable
+
OR
+
Digilent JTAG_SMT2_NC(On board)
JTAG Chain Block Diagram
H.Liu
VM1802 - BOOT - Micro SD Card VM1802 BOOT Modes
Configured by 4 pins of bank 503: MODE[0:3] (a DIP4 switch used) Modes Supported by Versal: eMMC1, JTAG, OSPI, QSPI24, QSPI32, SD0 (3.0), SD1 (2.0 or 3.0), SelectMAP Modes implemented on board: Dual QSPI (OSPI), SD1 (3.0)
Micro SD Card BOOT IP4856CX65 level translator will be used
An adapter pcb have been designed, and is being fabricated Same pinout as the one used by Xilinx EVM
Bidirectional translation between fixed 1.8V to 1.8V or 2.9V (SD side) SD power supply resettable by SD1_BUSPWR and POR_B
11
PMC 501 MIO # SD Signal26 CLK_IN27 DIR_1_328 CD29 CMD_H30 DATA031 DATA132 DATA233 DATA334 SEL35 DIR_CMD36 DIR_050 BUS_PWR51 n.c.
To SD Card VDD
H.Liu
VM1802 - BOOT - Dual QSPI
QSPI32 BOOT (dual-parallel) QSPI32: 32 refers to 32-bit address, search
offset limit is 8 Gb Two QSPI flash: MT25QU02GC
2 x 2Gb: 256M x 8 133 MHz VDD 1.8V
Reset signals 3 x VM1802 HDIO 1.8V GPIO pins One pin reset two flash (default) or
separate reset
12
PMC 500 MIO # QSPI Flash 1 QSPI Flash 20 CLK1 DQ12 DQ23 DQ34 DQ05 CS_B7 CSB8 DQ09 DQ1
10 DQ211 DQ312 CLK
* Reference: AM011, page 436, table 154
H.Liu
VM1802 - DDR4 Memories
3 x DDR4 Mini-DIMM w/ ECC slot implemented DDR4 #A/B/C: XPIO Bank 700 - 702/703-705/709-711, SPI I2C ADDR: 0x51-0x53
Mini-UDIMM DDR4 w/ ECC 16GB module purchased inodisk M4M0-AGS1YCSJ (PC4-2400), 32 GB sticks will be available in the future
One Si570 for each triplet (three banks) One TPS51200 to generate VTT for each slot Signal connection reference to the DDR4 mapping generated by a spreadsheet provided by Xilinx
13
Versal
DDR4 #A DDR4 #B
DDR4 #C
H.Liu
VM1802 - PS Gigabit Ethernet Circuitry references VMK180 TI DP83867 PHY Transceiver
RGMII interface Three power rails
VDDIO: SYS18 VDDA2P5: SYS25 VDD1P0: generated by MAX8869 LDO from SYS33
3 x Reset Signals Push button POR_B PMC_MIO48_501
14
LPD 502 MIO # RGMII Signals0 GTX_CLK1 TX_D02 TX_D13 TX_D24 TX_D35 TX_CTRL6 RX_CLK7 RX_D08 RX_D19 RX_D2
10 RX_D311 RX_CTRL24 MDC25 MDIO
H.Liu
VM1802 - Transceiver Links
16 links to FMC Direct connection
15
GTY Quad:CH LinkQ103:L0 FMC DP0Q103:L1 FMC DP1Q103:L2 FMC DP2Q103:L3 FMC DP3Q104:L0 FMC DP4Q104:L1 FMC DP5Q104:L2 FMC DP6Q104:L3 FMC DP7Q105:L0 FMC DP8Q105:L1 FMC DP9Q105:L2 FMC DP10Q105:L3 FMC DP11Q106:L0 FMC DP12Q106:L1 FMC DP13Q106:L2 FMC DP14Q106:L3 FMC DP15
GTY Quad:CH LinkQ202:L0 PCIe LAN 0Q202:L1 PCIe LAN 1Q202:L2 PCIe LAN 2Q202:L3 PCIe LAN 3Q203:L0 PCIe LAN 4Q203:L1 PCIe LAN 5Q203:L2 PCIe LAN 6Q203:L3 PCIe LAN 7Q204:L0 PCIe LAN 8Q204:L1 PCIe LAN 9Q204:L2 PCIe LAN 10Q204:L3 PCIe LAN 11Q205:L0 PCIe LAN 12Q205:L1 PCIe LAN 13Q205:L2 PCIe LAN 14Q205:L3 PCIe LAN 15
GTY Quad:CH LinkQ200:L0 Firefly Link 0Q200:L1 Firefly Link 1Q200:L2 Firefly Link 2Q200:L3 Firefly Link 3Q201:L0 Firefly Link 4Q201:L1 Firefly Link 5Q201:L2 Firefly Link 6Q201:L3 Firefly Link 7Q206:L0 Firefly Link 8Q206:L1 Firefly Link 9Q206:L2 Firefly Link 10Q206:L3 Firefly Link 11
16 links to PCIe FPGA to PEX: 0.22 uF AC for TX and RX PEX to edge: GTY TX: 0.22 µF AC, GTY RX: direct connection
12 links to Firefly Direct connection
H.Liu
VM1802 - Dual PCIe Gen4 x 8
GTY Banks for Dual PCIe Gen4 x 8: 202, 203, 204, 205 VM1802 PCIe endpoint location: 202, 205 (can be configured as PCIe Gen4 x 8)
Broadcom PCIe switch PEX88032 added PCIe REF CLK from edge connector is buffered by SI53156 (SCC clock tolerant)
4 x outputs to the GTY_REFCLK of 4 different GTY quads, 10 nF AC couple in between 1 x output to VM1802 XPIO bank 711 (10 nF AC, and biased to 0.6V)
Control signals Level translator between 3.3V and 1.8V (HDIO bank VCCIO) WAKE_B (from VM1802): HDIO.Bank306.H37 PERST_B (to VM1802): HDIO.Bank306.J35 PWRBRK (to VM1802): HDIO.Bank306.J34
SMBus translated to 1.8V (Board I2C master) by TCA9617A PRSNTN x1/x4/x8/x16 configurable by a 2 x 4 header connector PERST connected to FPGA PS MIO, FPGA PL, and PEX PWR_ON_RST
16
H.Liu
VM1802 Decouple Capacitors Decouple capacitor implementation is referenced to the VMK180 and the XPE back from Xilinx All ceramic capacitors Capacitor size, temp. coef. voltage rating
330 µF: 1210, X5R, 4V 100 µF: 0805, X5R or X6S, 4V 47 µF: 0805, X5R or X6S, 4V 10 µF: 0402, X5R or X6S, 6.3V 4.7 µF: 0402, X5R or X6S, 6.3V 1 µF: 0402 (size TBD), X5R or X6S, 6.3V (under FPGA BGA pins)
17
PWR Rail 330 µF 100 µF 47 µF 10 µF 4.7 µF 1 µFVCCINT, VCC_RAM 2 4 20 12
VCC_SOC, VCC_PMC, VCC_PSLP, VCC_PSFP 2 2 7 9
MGTYAVTT 2MGTYAVCC 2
VCCO per bank 1 1
H.Liu
FMC+ Expansion Connector
Signals Connected DP Links
DP[0:15] to 16 x GTY transceiver TX/RX pairs GBTCLK[0:2]_M2C to GBT_REFCLK of GTY quads
Differential pairs LA[0:33] to XPIO bank 709, 710 (VCCO = 1.5V) CLK[0:1] to XPIO bank 709, 710 clock capable pins
Two present signals Pull-up to 3.3V PRSNT_M2C: FMC mezzanine present signal, control JTAG TDI to TDO shunt HSPC_PRSNT_M2C: FMC+ mezzanine present signal
JTAG Already converted to 3.3V TDI to TDO shunt controlled by FMC PRSNT signal, also could be shunt with jumper resistor
Power: all monitored by INA226 via a 1 mΩ sensor resistor 12VP0V (5 pins) : board 12V input 3P3V (6 pins): SYS33 VADJ (4 pins): SYS15
I2C: Board I2C MUX CH2, 3.3V
18
H.Liu
Board I2C Bus Management TI TCA9548A for I2C Multiplexer Three I2C masters connected to the main I2C port
PCIe SMBus (3.3V to 1.8V) FTDI USB - I2C (3.3V to 1.8V) VM1802 HDIO (1.8V)
8 x Sub I2C Bus: 7 x 3.3V bus: FIREFLY_I2C, DDR4_I2C, FMC_I2C33, INA226_I2C, I2C33[A,B,C] 1 x 1.8V bus: I2C18
19
H.Liu
PCB Stack-Up
20
Material: Megtron7(N) 4 kinds of controlled impedance (single-ended: 50, differential: 85, 95, 100) Minimum via: 8mil/18 mil 1 kinds of back-drill: L24-L21 for SIG7, backdrill-bit: 18 mil, 6 mil(SCC)/8mil(TTM) clearance to via
No differential routing near backdrill via, so both SCC and TTM are able to produce our boards
H.Liu
Placement - Top
21
PEX88032 Clock Firefly DC/DCDC/DCLTM4678
RJ45/PHY
FMCP QSPI JTAG
H.Liu
Placement - Bottom
22
SD30 ADP
SD Card USB-JTAG
H.Liu
Routing Summary
Routing of the 3 x DDR4 Followed the impedance, trace delay, skew requirement on UG863 (Versal PCB design guideline)
GTY Routing PCIe:
Impedance: 85-ohm to edge, 100-ohm between FPGA and PEX Layer: layer 1, 14, 20, 22
Firefly (100-ohm): layer 14, 20, 22 FMC DP(100-ohm): layer 14, 22 P/N skew < 5 mil for PCIe 85-ohm, <3 mil for others
FMC LA Bus Match length to 6 inch
Other length matched data groups: SD 3.0, GbE23
H.Liu
Summary and Status
The design was finished in October, then has gone through internal reviews and revisions
The gerber files have been submitted to SCC for fabrication in November, all EQs have been resolved
Fabrication started this week due to the long lead-time of M7(N) material (will try halogen free material in next iteration), the estimated shipping date is 1/12/2021
Components have been purchased, all have been delivered We expect to get the assembled board in early february then start the testing
24
