[email protected] 12 October 2013
High Speed Links
1. High speed links in LHC and commercial applications2. On-going common projects for LHC Phase I upgrades3. Towards HL-LHC4. Conclusions
F. Vasey, CERN, PH-ESE
1. High Speed Links in LHC
22 October 2013
CPU
CPU
CPU
CPU
CPU
CPU
Switching network
Front-end DAQ interfaceFront-end DAQ
interfaceFront-end DAQ interfaceFront-end DAQ
interface 1
Front-end DAQ interface N
Front-end DAQ interface 2
Timing/triggers/sync Control/monitoringTrigger
1.1 Many different Link types Readout - DAQ:
Unidirectional Event frames. High rate Point to point
Trigger data: Unidirectional High constant data rate Short and constant latency Point to point
Detector Control System Bidirectional Low/moderate rate (“slow
control”) Bus/network or point to
point Timing: Clock, triggers,
resets Precise timing (low jitter
and constant latency) Low latency Fan-out network
(with partitioning)
3
Different link types remain physically separate, each with their own specific implementation
2 October 2013
1.2 For instance: Links in ATLAS
42 October 2013
Detector Purpose Media Dir. Rate Mbits/s
Quantity Comment
Pixel TTC/DCS/DAQ
Optical Down/up
40 (80) 250 Custom
SCT TTC/DCS/DAQ
Optical Down/up
40 8200 Custom
TRT TTC/DCS LVDS-Cu
Down 40 400 Custom
DAQ/DCS LVDS – Optical
Up 40 –1600 400 GOL
Ecal TTC Optical Down 80 TTC link
DCS LVDS-Cu
Down/Up
SPAC
DAQ Optical Up 1600 1600 Glink
Trigger Copper
Up Sub-det. Analog
Hcal TTC Optical Down 80 TTC link
DCS Copper
Down/up
CAN
DAQ Optical Up 1600 512 Glink
Trigger Copper
Up Analog
CSC, RPC, TGC DAQ Optical Up 1600 1200 Glink
MDT DAQ Optical Up 1600 1200 GOL
CSC TTC Optical Down 1600 200 Glink
CSC, RPC, TGC, MDT
DCS Copper
Down/up
CAN
RPC Trigger Optical Up 1600 Glink
1.3 For instance: Links in CMS
52 October 2013
Detector Purpose Media Dir. Rate Mbits/s
Quantity Comment
Pixel - strip TTC/DCS Optical Down/up
80 3.250 CCU, Custom
DAQ Optical Up 40Msamp~320
40.000 Custom analog
Ecal TTC/DCS Optical Down/up
80 3.000 CCU, Custom
DAQ Optical Up 800 3.000 GOL
Trigger Optical Up 800 5.300 GOL
Hcal TTC Optical Down 80 160
DCS Electrical
Down/up
n/a 160
DAQ & Trig Optical Up 1600 2.000 GOL
Muons TTC Optical Down 80 240 DT
DCS Optical Down/up
80 150 RPC
DAQ Electrical
Up parallel 800 DT
Trigger Optical Up 1.600 600 RPC
Trigger Electrical
Up parallel 800 DT,CSC
[email protected] 62 October 2013
1.4 High Speed Optical Links in LHC
Large quantity o(100k), large diversity Majority running @ o(1Gbps) From full custom to qualified COTS
Tuned to specific application and environment Developed by independent teams
Successful adoption of technology in HEP
[email protected] 72 October 2013
1.5 High Speed Links in LHC: Lessons Learned
Increase Link Bandwidth amortize system cost better
Share R&D effort use limited resources optimally
Strengthen quality assurance Programs Identify problems early Test at system level
Joint ATLAS/CMS NOTE ATL-COM-ELEC-2007-001
CMS-IN-2007/066 https://edms.cern.ch/document/882775/3.8
[email protected] 82 October 2013
1.6 High Speed (short distance) Links outside LHC
Rapid progress driven by: Successful standardization effort
100 GbE standard ratified in 2010 by IEEE Availability of hardware cores embedded in FPGAs
50+ x 10G transceivers in modern FPGAs Commercial availability of MSA-based hardware
Commodity 10G and 40G devices Emerging 100G and 400G engines
Current LAN rates @ o(10Gbps), ramping up to 40Gbps
Widening performance gap compared to HEP But consider:
Specific environmental constraints Long development time: vintage 2000 hardware in LHC
[email protected] 92 October 2013
2. On-going Development Projects for LHC Phase I Upgrades
Initially aiming at a single SLHC target Launched in 2008 Timely developments for phase I upgrades
Working Groups Microelectronics User Group (MUG) Optoelectronics Working Group (Opto WG)
Common Projects Rad Hard Optical Link
GigaBit Transceiver (GBT) project (chip-set) & GBT-FPGA project
Versatile Link (VL) project (opto) & Gigabit Link Interface Board (GLIB)
Many others …
[email protected] 102 October 2013
2.1 Rad Hard Optical Link Common Project
Requirements: General
Bi-directional High data rate: 4.8Gbits/s Low and constant latency (for TTC and trigger data paths) Error detection and correction
Environment Radiation hard ASICs (130nm) and radiation qualified opto-electronics at
Front-End Magnetic Field tolerant devices at Front-End Flexible chip interface (e-links) to adapt to various architectures Compatibility with legacy fibre plants
Back-end COTS High-end FPGAs with embedded transceivers
GBT-FPGA firmware Parallel optics
Commitment to deliver to ATLAS, CMS and LHCb in 2014-2015
2.2 Impact on System Architecture Custom development for
difficult Front-End Firmware only for COTS-
based Back-End Evaluation platform for
system-tests
CPU
CPU
CPU
CPU
CPU
CPU
Switching network
Front-end interface
Front-end interface
Front-end interface
Timing/triggers/sync
Control/monitoring
Trigger
DCS network
Rad-hard optical links
2 October 2013
[email protected] 122 October 2013
2.3 GBT Project Status Project started in 2008
GBT (Serializer/Deserializer) 1st iteration (GBT-Serdes) in
2009 2nd iteration (GBTx) in 2012 Packaging in 2013 3rd iteration and prod in 2014
GBLD (Laser Driver) Final iteration (V4.1/V5) in 2013
GBTIA (Pin Diode Receiver) Final iteration (V3) in 2012
GBTSCA (Slow Control ASIC) Final version expected in 2014
GBT-FPGA firmware Available Tracking major FPGA families
Project delivers Chipset for Front-End GBT-FPGA Back-End firmware
[email protected] 132 October 2013
2.4 VL Project Status Kick-off: April08 Proof of concept: Sep09 Feasibility demo: Sep11 Project delivers
Custom built Rad Hard VTRx Production readiness: Apr14 Early delivery of rad-soft VTTX to
CMS-Cal-Trig: Dec13 Recommendations for
Fibre and connectors Backend optics
Evaluation Interface boards (GLIB) Experiments
Design their own system Select passive and backend components
based on VL recommendations and on their own constraints
2.5 Packaging and Interconnects Status
GBT 20x20 BGA with on-
package crystal and decoupling capacitors
CERN<>IMEC<>ASE 5 iterations to freeze
design 1-4 weeks per iteration 6 months to first prototype Mask error, re-spin,
+2months
VL High speed PCB simulation
and design Injection-moulded ULTEM
2xLC connector latch and pcb support
Prototyping started 2009, moulded parts delivered 2013
2 October 2013
[email protected] 152 October 2013
2.6 Rad Hard Optical Link Project Effort
6 years of development Launched in 2008 Delivery in 2014-15
6 institutes involved CERN, FNAL, IN2P3, INFN, Oxford, SMU Estimated 80 MY + 2-3 MCHF development
effort
[email protected] 162 October 2013
3. Towards HL-LHC
Higher Data-rate Lower Power Smaller Footprint Enhanced Radiation Resistance
Not to be forgotten: Fast electrical links Radiation-soft links
Not all in same link
[email protected] 172 October 2013
3.1 Higher Data-Rate and Lower Power
Requires migrating to a more advanced technology node: 65nm
Qualify technology Establish stable support framework and design tools for full
duration of development Design new ASICs taking advantage of technology
Either high speed (multiply by two) Or low power (divide by four)
VL Opto components 10G capable But will need to qualify new emerging components
Electrical interconnects and packaging become performance limiters
Build up expertise Train with relevant simulation and design Tools Establish relationships with selected suppliers
[email protected] 182 October 2013
3.2 Smaller Footprint
GBT package size can be shrunk by limiting the number of IO pads and going to fine pitch BGA
Will affect host board design
VTRx concept has been pushed to its size limit: SF-VTRx
Not sufficient for some tracker layouts
Tracker frontends will need custom packaging
Industry to be approached
VTRx
SF-VTRx
[email protected] 192 October 2013
3.3 Enhanced Radiation Resistance
ASICs likely to be OK
Active opto devices tested up to now do not resist fluences beyond 1016 cm-2
Still OK for HL-LHC Tk, not for pixels
Tight margins Are there
alternatives for fluences beyond 1016 cm-2 ?
Reconsider Passives?
modulators
Tx
Rx
HL-LHC TK
HL-LHC TK
[email protected] 202 October 2013
3.4 Si-Photonics, a paradigm changing technology?
Si is an excellent optical material with high refractive index (but indirect
bandgap) Is widely available in high quality grade Can be processed with extreme precision using deep submicron CMOS
processing techniques So, why not build a photonic circuit in a CMOS Si-wafer?
[email protected] 212 October 2013
3.5 Si-Photonics, status in the community
Commercial devices tested at CERN and ANL
Excellent functional performance Moderate radiation resistance
limited by controller ASIC failure On-going collaborations with
one academic and two industrial partners
Simulation tools in hands Selected building blocks
under test No usable conclusion so far,
much more work needed Packaging is challenging
Assess radiation hardness first ! Luxtera QSFP+ Si-Photonics chip
[email protected] 222 October 2013
3.6 Not to be forgotten
Fast electrical data links are not obsolete !!! Short distance, on-board serial links Aggregation to high speed opto-hubs Very high radiation fields (HL-LHC pixels) Develop expertise and tools
Detectors with modest radiation levels may not need custom front-ends Qualify COTS and/or Radiation-soft components Shortlist recommended parts Continuously track market evolution
23
4. Conclusions High speed links are the umbilical cord of the experiments Meeting the HL-LHC challenge will require:
Qualifying new, emerging technologies Designing electronics, interconnects, packages and perhaps even
optoelectronics Maintaining expertise, tools and facilities Investing with a few selected industrial partners
The community is healthy, but small and fragmented Working groups and common projects put in place for phase I
upgrades are effective and should be continued in phase II Additional projects and working groups could be created
WG on fast electrical links & signal integrity, and radiation-soft links Project on Si-photonics for HEP applications
Manpower is the real bottleneck, time is the contingency Development time remains very long
6 years, 6 institutes, 80 MY were required to reach production readiness for phase I
Widening gap with industry
Development
Service
Commercial liaison
Design
Common projects
Working groups
People
