WBS 6.7.4 Readout Interface firmware
1
Alexander Paramonov Level-3 Manager & CAM
Argonne National Laboratory
U.S. ATLAS HL-LHC Upgrade Project DOE CD-1 Director’s Review
Brookhaven National Laboratory Upton, New York May 2018 (DOE)
Deliverables
The global theme of the WBS is interfacing custom and commodity systems.
6.7.4.1: Firmware for Hardware Track Trigger Interface (HTT IF) to transmit data between HTT and the processor farm
6.7.4.2: Firmware for the Global Trigger to transmit data between the Global Trigger and FELIX
Both systems are new additions to the TDAQ architecture.
6.7.4.2
6.7.4.1
2
Deliverable 6.7.4.1
The hardware track trigger (HTT) are key to operating of the ATLAS experiment during the HL-LHC program The regional tracking will reconstruct tracks with pT>2 GeV at 1 MHz in
selected regions of interest.
The global tracking will reconstruct all tracks with pT>1 GeV at 100 kHz.
The HTT will be connected to commodity data networks through the HTT interface (HTT IF).
We will need to develop data handling firmware that resides in the HTT IF.
3
Mainboard
(UPenn)
Optical link HTT Interface Network (Ethernet) Custom data link Event Filter
(Comp. farm)
Deliverable 6.7.4.1
Technical challenges Data routing is not trivial because significant data duplication is needed
between the track processor boards in each ATCA shelf.
o We defined specifications for the data distribution architecture and i/o bandwidth. The spec review is in June 2018. The work is uncosted (AP).
We need to avoid large time skew when the data is delivered from HTT IF to the track processor boards.
o I demonstrated that the system would be much cheaper if we use ATCA backplane to share pixel and strip hits. That requires low time skew.
4
Deliverable 6.7.4.1
International collaboration All the firmware for the HTT IF will be developed in the US.
No hardware needs to be produced under this WBS. Each HTT IF server will need 2 HL-LHC FELIX PCIe cards built under WBS 6.7.3.
I serve at the L3 ATLAS manager for the HTT IF and infrastructure.
I am an edition of the HTT IF specification document.
5
Deliverable 6.7.4.2
The global common module is described in WBS 6.7.1
Argonne will develop firmware to connect the global trigger system and FELIX. Most of data flow is from Global Event Processors into FELIX.
Monitoring data from MUX’es.
TTC will be provided by FELIX to all the boards of the Global trigger.
6
Deliverable 6.7.4.2
The firmware blocks that needs to be developed is depicted with the green boxes. The data format and protocols need to be defined.
The TTC is key to readout.
7
Deliverables/WBS dictionary
The BoE of Readout Interface Firmware covers two firmware items, Tracking Data Provider Firmware (6.7.4.1), and Global-to-FELIX Firmware (6.7.4.2).
6.7.4.1: The WBS covers the development of firmware for the HTT Interface (HTTIF) to prepare and transmit data between the Dataflow network and the HTT main boards in both directions. The Hardware Track Trigger (HTT) will reconstruct tracks for the HL-LHC ATLAS Event Filter (EF) system.
6.7.4.2: The HL-LHC ATLAS GLOBAL Trigger will send trigger output data and monitoring data to the FELIX system. This WBS covers the development of firmware to interconnect the Global Common Modules (GCM) and the readout system (FELIX). The firmware will prepare the GLOBAL trigger output data and monitoring data to the FELIX system, and will handle the communication between the GCM and the FELIX.
8
Copied from the BoE.
Technical Specifications
See Jinlong’s talk
9
Ongoing work
Development of the HTT IF specification is actively ongoing. Argonne and UPenn are collaborating since the I/O has major impact
on both the HTT IF and the track processor boards that UPenn is developing.
The initial bandwidth estimated were developed and included into the ATLAS TDAQ TDR.
Now we are refining the bandwidth calculations by using the recent layouts of the inner tracker (ITk).
We have prepared a draft of data format for HTTIF-to-TP communications and HTT IF firmware architecture
The needed FELIX interfaces for the Global Trigger System have been outlined in the TDR. Well understood architecture.
The bandwidth is less of a challenge than for the HTT IF.
10
R&D
Next steps: Finalize the data transmission protocols
Develop demonstrator firmware to evaluate the FPGA utilization and performance of the circuits.
11
Project Management
IC for 6.7.4.1 = Alexander Paramonov
IC for 6.7.4.2 = Jinlong Zhang
Weekly meetings with ATLAS colleagues
I am involved in the ATLAS HTT coordination
12
ES&H
Safest is of the highest priority within the project Work at Argonne National Laboratory dheres strictly to the ES&H
policies:
Our ES&H officers:
o periodically inspect out laboratories
o provide oversight and guidance
o Interact with CERN ES&H
Main hazards for the WBS Electrical testing
o Low voltages (below 15V DC)
All work is done in compliance with safety policies at ANL and CERN 13
Institute Institute ES&H Contact
ANL/Directorate Urs Geiser (https://www.anl.gov/safety)
ANL/HEP Leon Reed
Cost and Schedule Estimate Methodology
Analogy FELIX for the LS2 upgrade and FTK FLIC firmware (FTK= fast track
trigger)
Expert Opinion Consulted an experienced EE to get the estimates of effort and
duration for the tasks
14
Schedule 6.7.4.1
Demonstrator phase CY18-20 Preliminary design review
Specifications for HTT IF firmware/functionality and the data formats
Design demonstrator firmware for the HTT IF
o Evaluate the FPGA resource utilization, system performance, etc.
Prototyping Phase: CY20-21 Final design review
Prototype firmware.
System integration at CERN starts. The integration will be done by ATLAS.
System integration phase CY21-24 Production readiness review The data transmission firmware will be finalized
System integration at CERN will continue
Needed at CERN by 7/1/2015.
15
Schedule 6.7.4.2
Demonstator phase: CY19-20 Initial design review
Complete specifications for GEP and MUX I/O to FELIX
Design demonstrator data transmission firmware
Evaluate the FPGA resource utilization and performance with demonstrator GCM boards.
Testing will be done at BNL.
Prototyping Phase: CY20-21 Final design review
The data transmission firmware will be fully prototype.
Will continue traveling to BNL for testing.
System integration at CERN starts
System integration phase CY21-24 Production readiness review Firmware finalized by February of 2024.
System integration will continue and extensive tests will be needed to finalize the firmware.
Needed at CERN 09/02/2015
16
Schedule
Integration will be a challenge because of the large number of components involved and the number of participating institutions. Utilization of the FPGA resources and performance needs to be
understood early.
Track processing requests comes from a number of event filter nodes.
Each HTT IF server handles I/O from 28 track processing boards.
The data flow (back pressure) needs to be propagated through the entire chain of processing elements.
17
Budget and Effort
18
92%
1%
7%
Labor, Material and Travel (%)
BCWS Labor
BCWS Material
BCWS Travel
6.07.04.01 Tracking Data Provider Firmware 140,704 76,743 57,348 104,276 165,273 123,409 667,754
ENG 118,750 76,743 51,874 90,326 157,508 109,053 604,255
MAT 16,794 16,794
TRAVF 5,160 5,474 13,950 7,765 14,356 46,705
6.07.04.02 Global-to-FELIX Interface 79,560 125,062 109,493 105,597 143,775 563,488
ENG 75,124 123,459 106,390 96,670 129,211 530,855
TRAVD 519 1,602 3,103 5,224
TRAVF 3,917 8,927 14,564 27,408
Grand Total 140,704 156,303 182,410 213,769 270,870 267,184 1,231,241
Cost Profile
19
FY19 FY20 FY21 FY22 FY23 FY24
BCWS
ENG 118,750 151,867 175,334 196,716 254,178 238,264
MAT 16,794
TRAVD 519 1,602 3,103
TRAVF 5,160 3,917 5,474 13,950 16,692 28,920
0
50,000
100,000
150,000
200,000
250,000
300,000
Total Cost by Resource Category
FTE Profile
20
FY18 FY19 FY20 FY21 FY22 FY23 FY24 FY25
Y-FTE
ENG 0.58 0.62 0.64 0.74 0.98 0.85
UNCOST 0.20 0.20 0.20 0.20 0.20 0.20 0.20 0.20
0.00
0.20
0.40
0.60
0.80
1.00
1.20
1.40
Axi
s Ti
tle
FTE by Resource Category
Risks
From the risk registry
21
6.7.4 Readout Interface Firmware
Threat RD-06-07-
04-001 Active 19-Apr-18 31-Aug-22
Different I/O interface for hardware track trigger (HTT)
The different throughput and latency requiremets on I/O may reuqire different firmware functionality and performance on HTT Interface
6.7.4 Readout Interface Firmware
Threat RD-06-07-
04-002 Active 19-Apr-18 30-Nov-22
Additional data handling needed between GLOBAL and FELIX
More functionalities may be needed to communicate between GLOBAL and FELIX for data transmission
Resource optimization
AAA
22
External Dependencies
None
23
Milestones
24
6.07.04.01 Tracking Data Provider Firmware FTK710580EX ATLAS External Milestone: Preliminary Design Review 05/14/19 5
FTK710620R REQD: HTT IF Demonstrator Main Board from UPenn 10/01/18 6
FTK710710R REQD: demonstrator PCIe I/O card (FELIX) 09/11/20 6
FTK710760R REQD: Prototype HTT main board available 03/17/21 6
FTK710780R REQD: Prototype 1 of the PCIe I/O card 09/27/21 6
FTK710840 COMP: Prototype HTT IF firmware complete 06/13/22 4
FTK710870EX ATLAS External Milestone: Final Design Review 08/03/22 5
FTK710900R REQD: Prototype 2 of the PCIe I/O card (FELIX Proto 2) 05/31/22 6
FTK710920R REQD: Preproduction version of the HTT main board 10/10/22 6
FTK710940 COMP: Pre-production HTT IF firmware complete 05/17/23 4
FTK710970EX ATLAS External Milestone: Production readiness review 06/15/23 5
FTK710980R REQD: Preproduction version of the PCIe I/O card 06/16/23 6
FTK711010R REQD: Production version of the PCIe I/O card 07/06/23 6
FTK711040R REQD: Production version of the HTT main board 12/11/23 6
FTK711070 COMP: Production HTT IF Firmware 03/19/24 4
6.07.04.02 Global-to-FELIX Interface RID3005R REQD: Shcmatics of the prototype-1 GCM 12/24/19 6
RID3025EX External Milestone: ATLAS Preliminary design review for the GCM 05/27/20 5
RID3045R REQD: prototype-1 GCM board 08/18/20 6
RID3075R REQD: Shcmatics of the prototype-2 GCM 07/01/21 6
RID3085R REQD: ATLAS Specification for the trigger and aggregator firmware 07/27/16 6
RID3095R REQD: Prototype-2 GCM board Completed 10/26/21 6
RID3120R REQD: Shcmatics of the prototype-3 GCM 07/13/22 6
RID3145EX External Milestone: ATLAS Final Design Review 11/10/22 5
RID3150R REQD: prototype-3 GCM board 02/01/23 6
RID3230EX External Milestone: ATLAS Production readiness review 10/13/23 5
RID3260 Milestone: Readout firmware complete 05/16/24 4
Closing remarks
We have established a technical plan for the WBS 6.7.4 Readout Interface Firmware The design is based on our experience with FTK (FLIC) and FELIX
The schedule is loaded and the risks are assigned
The schedule is well aligned with the ATLAS effort.
We are ready for CD-1
25
Backup
26
Resources
Argonne has an experienced EE design team.
Recent projects Firmware for FELIX
FLIC for FTK (firmware and boards)
LVPS for Tile Cal (rad-tolerant circuitry)
Front-end electronics for Tile Cal Upgrade (QIE12-based PMT readout)
Photodetector readout for proto DUNE
Trolley motor control and B-field sensor readout for g-2
27
Who I am?
Served as L2 for Tile R&D
Coordinator of ATLAS center at Argonne
Chair of the US ATLAS speakers committee
Completed EVMS training in 2016
ATLAS Responsible/contact person for HTT I/O and infrastructure
Hardware experience
Designed firmware for time-to-digital converter boards for CDF: data preparation for eXtremely Fast Trigger. Developed software and tested the boards.
Designed firmware for control and readout of QIE front-end cards for ATLAS Tile Cal. Designed software. Evaluated the system with testbeams.
Evaluation of Si photonics fiber-optical data links for radiation tolerance: tested BER with PRBS at 10 Gbps and the IC-specific slow controls.
Previous experiments: H1 (DESY) and CDF (FNAL)
28
Data distribution for HTT
The input bandwidth each Track Processor (TP) board needs is much greater than the total HTT input bandwidth divided by the number of boards.
We have evaluated the data duplication and the results are in the TDAQ TDR.
There are 576 Associative memory TPs and 96 Second stage TPs in the system.
eta
phi ~Beam size in z/R(module)
~R/pT
Layer 13 - SCT
Red box = coverage of the track reconstruction
Blue box and points = modules needed for the
track reconstruction
29
From the TDAQ TDR
Data distribution for HTT
30
