Andrew J. Lankford University of California, Irvine

Trigger & Data AcquisitionTrigger & Data Acquisitionfor LHC Upgradesfor LHC Upgrades

International Workshop on Future Hadron CollidersInternational Workshop on Future Hadron CollidersFermilabFermilab

October 16-18, 2003October 16-18, 2003

Andrew J. LankfordAndrew J. LankfordUniversity of California, IrvineUniversity of California, Irvine

17 Oct 200317 Oct 2003 Lankford – Trigger & Data AcquisitionLankford – Trigger & Data Acquisition

AcknowledgementsAcknowledgements

My presentation draws heavily upon:My presentation draws heavily upon:• Work performed by contributors to the Work performed by contributors to the

preliminary studies reported in:preliminary studies reported in:Physics Potential and Experimental Challenges of Physics Potential and Experimental Challenges of the LHC Luminosity Upgrade the LHC Luminosity Upgrade (hep-ph/020487).(hep-ph/020487).

• A recent presentation by Nick Ellis at Erice.A recent presentation by Nick Ellis at Erice.


SLHC: Implications of Higher LuminositySLHC: Implications of Higher Luminosity

Higher Luminosity =>Higher Luminosity =>• Increased detector occupancyIncreased detector occupancy• Increased trigger ratesIncreased trigger rates• Increased radiation effectsIncreased radiation effects

Assumed SLHC luminosity = 10Assumed SLHC luminosity = 103535 cm cm-2-2ss-1-1


SLHC: Implications of Higher Luminosity - SLHC: Implications of Higher Luminosity - 22

Higher Luminosity => Increased detector occupancyHigher Luminosity => Increased detector occupancy• Assuming 12.5 ns bunch spacing,Assuming 12.5 ns bunch spacing,

125 interactions/crossing125 interactions/crossing– Occupancy of tracks increases 5-fold Occupancy of tracks increases 5-fold (10-fold for some sub-detectors)(10-fold for some sub-detectors)

Radiation-induced hits increase 10-fold.Radiation-induced hits increase 10-fold. Pile-up noise increases 2.2-fold Pile-up noise increases 2.2-fold (3-fold for some sub-detectors)(3-fold for some sub-detectors)

• Pile-up degrades performance of trigger algorithmsPile-up degrades performance of trigger algorithms Reduction in efficiency of e/gamma isolation cutsReduction in efficiency of e/gamma isolation cuts Increased muon candidate rates due to radiation-induced Increased muon candidate rates due to radiation-induced

accidentalsaccidentals• Larger event size to read outLarger event size to read out

Increased by factor between 5 and 10Increased by factor between 5 and 10 Demands reduced trigger rate or increased data bandwidthDemands reduced trigger rate or increased data bandwidth



Higher Luminosity => Increased trigger ratesHigher Luminosity => Increased trigger rates• Arising from:Arising from:

Increased interaction ratesIncreased interaction rates Occupancy/pile-up induced trigger degradationOccupancy/pile-up induced trigger degradation

– Less rejection at fixed efficiencyLess rejection at fixed efficiency• Need more selective triggers for same trigger rateNeed more selective triggers for same trigger rate

Increased thresholdsIncreased thresholds More exclusive selection More exclusive selection (less inclusive trigger)(less inclusive trigger) Fortunately, more selective triggers are okayFortunately, more selective triggers are okay

– (see later)(see later)• Need even more data acquisition bandwidth if higher Need even more data acquisition bandwidth if higher

trigger rate.trigger rate.



Higher Luminosity => Increased radiation effectsHigher Luminosity => Increased radiation effects• Directly affects on-detector trigger logicDirectly affects on-detector trigger logic• Mechanisms:Mechanisms:

Permanent damagePermanent damage Single-event-upset effectsSingle-event-upset effects

• (Note that radiation effects upon detectors and their electronics (Note that radiation effects upon detectors and their electronics must be handled by trigger and by data acquisition.)must be handled by trigger and by data acquisition.)


Enabling TechnologiesEnabling Technologies Enabling technologies for readout and trigger:Enabling technologies for readout and trigger:

• Integrated circuitsIntegrated circuits Custom ICs, FPGAs, memories, etc.Custom ICs, FPGAs, memories, etc.

• Commodity computingCommodity computing processors, networking, memory, storage, fiberopticsprocessors, networking, memory, storage, fiberoptics

These technologies enabled current generation of expts.These technologies enabled current generation of expts.• Including the nearly-deadtimeless, integrated trigger Including the nearly-deadtimeless, integrated trigger

and data acquisition systems that are now standard.and data acquisition systems that are now standard. These technologies will also be the technologies that These technologies will also be the technologies that

enable successful upgrades & future experiments.enable successful upgrades & future experiments. R&D programs to develop these technologies for HEP R&D programs to develop these technologies for HEP

applications were crucial to current generation of expts.applications were crucial to current generation of expts. A vital R&D program will be crucial to SLHC upgrades.A vital R&D program will be crucial to SLHC upgrades.


Data Transfer & Data ProcessingData Transfer & Data Processing Trigger & Data Acquisition systems provide:Trigger & Data Acquisition systems provide:

• Data transferData transfer• Data processingData processing

Limitations or costs of these functions define limitations Limitations or costs of these functions define limitations and costs of Trigger/DAQ systems.and costs of Trigger/DAQ systems.• e.g. First Level Triggerse.g. First Level Triggers (FLT)(FLT)

Data input to FLT limits density of FLT electronics.Data input to FLT limits density of FLT electronics. Causes extensive interconnections betw. modules and Causes extensive interconnections betw. modules and

complex backplanes.complex backplanes.– Needed to connect steps in FLT selection.Needed to connect steps in FLT selection.– Needed to seamlessly find tracks and clusters in different Needed to seamlessly find tracks and clusters in different

sections of detector (to avoid loss of efficiencysections of detector (to avoid loss of efficiency


SLHC Trigger MenuSLHC Trigger Menu

Need for 3 types of triggers foreseen:Need for 3 types of triggers foreseen:• Discovery physicsDiscovery physics

Very high-Very high-pTpT (thresholds as high as hundreds of GeV)(thresholds as high as hundreds of GeV)

• Completion of LHC physics programCompletion of LHC physics program e.g.e.g. precise measurements of Higgs sector precise measurements of Higgs sector Lepton/photon/jet thresholds as low as for LHCLepton/photon/jet thresholds as low as for LHC Final states known => exclusive selection possibleFinal states known => exclusive selection possible

• Control / Calibration triggersControl / Calibration triggers e.g.e.g. W’s, Z’s, top W’s, Z’s, top Low thresholds needed, but can be pre-scaledLow thresholds needed, but can be pre-scaled

None of the above pose rate problems.None of the above pose rate problems.


Inclusive Triggers: samples & ratesInclusive Triggers: samples & ratesLHC SLHC

Selection Threshold Rate Threshold Rate (GeV) (kHz) (GeV) (kHz)inclusive single muon 20 4 30 25inclusive, isolated e/gamma 30 22 55 20†

muon pair 6 1 20 fewisolated e/gamma pair 20 5 30 5 inclusive jet 290 0.2 35 1jet + missing ET 100+100 0.5 150+80 1-2inclusive ET 150 <1multi-jet triggers various 0.4 various low

Note that inclusive e/γ trigger dominates rate. (†Added degradation from pile-up not included above)


FLT: FLT: Importance of FLT upgradesImportance of FLT upgrades

Upgrades to First Level Trigger Upgrades to First Level Trigger (FLT)(FLT) are of central are of central importance.importance.• They can reduce new demands on:They can reduce new demands on:

Front-end electronics Front-end electronics (FEE)(FEE) – Retaining 100 kHz FLT rate avoids changes to FEE Retaining 100 kHz FLT rate avoids changes to FEE

systems that do not require upgrade for other reasons.systems that do not require upgrade for other reasons. Data Acquisition Data Acquisition (DAQ)(DAQ)

– By reducing new demands for data bandwidth.By reducing new demands for data bandwidth. High Level Triggers High Level Triggers (HLT)(HLT)

– By reducing new demands for algorithms and processingBy reducing new demands for algorithms and processing FLT upgrades deserve early consideration.FLT upgrades deserve early consideration.


FLT: FLT: New Demands on ProcessingNew Demands on Processing

New demands on FLT processing posed by:New demands on FLT processing posed by:• Increased event pile-up & occupancyIncreased event pile-up & occupancy• New sub-detectors or readoutNew sub-detectors or readout• 12.5 ns crossing period12.5 ns crossing period

Some Some (at least)(at least) new processing will be needed. new processing will be needed.


FLT: FLT: Impact of 12.5 ns crossingsImpact of 12.5 ns crossings

LHC FLTs:LHC FLTs:• pipelined processors pipelined processors • driven at 40 MHz LHC crossing ratedriven at 40 MHz LHC crossing rate• identify crossing of interestidentify crossing of interest

Should SLHC FLTs run at 80 MHz Xing rate?Should SLHC FLTs run at 80 MHz Xing rate?• Can FLTs remain at 40 MHz ?Can FLTs remain at 40 MHz ?• Can FLTs work at both 40 & 80 MHz ?Can FLTs work at both 40 & 80 MHz ?


FLTFLT: : Can SHLC FLTs operate at 40 MHz ?Can SHLC FLTs operate at 40 MHz ?

Concept of readout Concept of readout ‘time frames’‘time frames’• FLT identifies pair of crossings FLT identifies pair of crossings (or more)(or more) for read out for read out• e.g. e.g. PEP II / BABARPEP II / BABAR

PEP II runs at 250 MHz PEP II runs at 250 MHz (4 ns).(4 ns). BABAR system clock = 60 MHz BABAR system clock = 60 MHz (16 ns).(16 ns). L1T runs at multiples of 16 ns.L1T runs at multiples of 16 ns. DAQ reads out time frames as large as few microsec, as DAQ reads out time frames as large as few microsec, as

appropriate to subdetector technology.appropriate to subdetector technology.

• May not be possible for many FEE systemsMay not be possible for many FEE systems• Increases data volume on FEE links & through DAQIncreases data volume on FEE links & through DAQ

Yes, SLHC FLTs could operate at 40 MHz,Yes, SLHC FLTs could operate at 40 MHz,• But, 40MHz FLTs will generally suffer more from pile-up.But, 40MHz FLTs will generally suffer more from pile-up.


FLTFLT: : Should SLHC FLTs operate at 80 MHz ?Should SLHC FLTs operate at 80 MHz ?

Advantages:Advantages:• Reduced pile-up effectsReduced pile-up effects

more effective algorithmsmore effective algorithms less data volume less data volume (for detectors that identify Xing during r/o)(for detectors that identify Xing during r/o)

• Ability to identify crossing that caused triggerAbility to identify crossing that caused trigger• Allows more processing steps within fixed latencyAllows more processing steps within fixed latency

• 80 MHz electronics feasible 80 MHz electronics feasible (Portions already at 80MHz.)(Portions already at 80MHz.)

Disadvantages:Disadvantages:• Requires FLT upgradesRequires FLT upgrades• Increased data bandwidth into & within FLTIncreased data bandwidth into & within FLT• FEE may not be able to deliver data to FLT at 80MHzFEE may not be able to deliver data to FLT at 80MHz

• Study Study (cost/benefit)(cost/benefit) needed. needed.


FLTFLT: : Can SLHC FLTs operate at both 40 & 80 MHz ?Can SLHC FLTs operate at both 40 & 80 MHz ?

Portions of FLT could operate at 40 MHz while Portions of FLT could operate at 40 MHz while other portions operate at 80 MHz.other portions operate at 80 MHz.

Note: Identification of 12.5 ns crossing can be Note: Identification of 12.5 ns crossing can be derived from 40 MHz samples for calorimeters.derived from 40 MHz samples for calorimeters.• Time resolution of calorimeter pulses is much better than 25 ns.Time resolution of calorimeter pulses is much better than 25 ns.• Timing of pulses is derived by digital filtering of multiple samples.Timing of pulses is derived by digital filtering of multiple samples.• Such a scheme already used in ATLAS CSCs Such a scheme already used in ATLAS CSCs w/ sampling at 20 MHzw/ sampling at 20 MHz

Some such hybrid likely to be a good solution.Some such hybrid likely to be a good solution.


High-Level Triggers & Data AcquisitionHigh-Level Triggers & Data Acquisition Commercial computing & networking technologies will provide the Commercial computing & networking technologies will provide the

advances necessary for High Level Triggers (HLTs) and Data advances necessary for High Level Triggers (HLTs) and Data Acquisition (DAQ) systems to perform at SHLC rates.Acquisition (DAQ) systems to perform at SHLC rates.• e.g. Moore’s Law will provide e.g. Moore’s Law will provide x10x10 improvement in price:performance ratio improvement in price:performance ratio wrtwrt

LHC start-up ~5 years after start-up.LHC start-up ~5 years after start-up.• e.g. Appetite for high-bandwidth graphical computing applications will drive e.g. Appetite for high-bandwidth graphical computing applications will drive

networking capabilities up and costs down.networking capabilities up and costs down.

R&D is necessary to develop technology advances for HEP applications.R&D is necessary to develop technology advances for HEP applications.

• Data transfer:Data transfer: Data linksData links HLT/DAQ networksHLT/DAQ networks Data Sources / Readout BuffersData Sources / Readout Buffers

• Data processing:Data processing: New HLT algorithms New HLT algorithms to maintain HLT selectivity with increased pile-upto maintain HLT selectivity with increased pile-up

• ““Complexity handling”Complexity handling”


Data LinksData Links

New challenges at SLHCNew challenges at SLHCfor data links from Front-end Electronics to Trigger & DAQfor data links from Front-end Electronics to Trigger & DAQ

• Higher bandwidthsHigher bandwidths due to higher occupancies due to higher occupancies (& FLT rates?)(& FLT rates?)

• Radiation effects at transmitting endRadiation effects at transmitting endfor some subdetectors, for some subdetectors, e.g.e.g. systems optimized for LHC systems optimized for LHC

• Also to increase input capabilities of FLTsAlso to increase input capabilities of FLTs Limitations on data input often limit FLT capability.Limitations on data input often limit FLT capability.

R&D neededR&D needed• Applications of commercial developmentsApplications of commercial developments• Possible custom developmentsPossible custom developments


HLT/DAQ NetworksHLT/DAQ Networks Networks connect HLT processors to data sources.Networks connect HLT processors to data sources.

• Every processor connects to every source.

• Data moves from sources to processors in parallel (“parallel event building”) to handle high trigger rates.

CMS/Cittolin


HLT/DAQ NetworksHLT/DAQ Networks Commercial networking equipment provides the Commercial networking equipment provides the

infrastructure for interconnection (switches) and data infrastructure for interconnection (switches) and data transfer (links).transfer (links).• At present, the cost of this equipment determines how At present, the cost of this equipment determines how

much data experiments can afford to move to HLT much data experiments can afford to move to HLT processors.processors. Thus, cost can limit trigger rate capability.Thus, cost can limit trigger rate capability. ATLAS has adopted an “RoI-based” Level 2 trigger in order ATLAS has adopted an “RoI-based” Level 2 trigger in order

to reduce overall data bandwidth requirements.to reduce overall data bandwidth requirements. CMS has developed a scaleable event building architecture.CMS has developed a scaleable event building architecture.

SLHC network bandwidth at least 5-10 times LHC b/w.SLHC network bandwidth at least 5-10 times LHC b/w.• Even if SLHC FLTs can provide same rate as at LHCEven if SLHC FLTs can provide same rate as at LHC• Network bandwidth requirements grow with occupancy.Network bandwidth requirements grow with occupancy.


HLT/DAQ NetworksHLT/DAQ NetworksComplete HLT/DAQ systems require large networks.

Individual network switches not yet of size required for full interconnectivity.


HLT/DAQ Networks R&DHLT/DAQ Networks R&D R&D should track evolution of network technologyR&D should track evolution of network technology

• Seek switches with requisite number of ports to avoid Seek switches with requisite number of ports to avoid multiple switches & extra portsmultiple switches & extra ports Technical challenge to manufacture of switches with many Technical challenge to manufacture of switches with many

high-speed ports is bandwidth capability of switch ‘fabric’ high-speed ports is bandwidth capability of switch ‘fabric’ (backplane) that interconnects ports.(backplane) that interconnects ports.

• Switches from different vendors behave differently Switches from different vendors behave differently within HEP systemswithin HEP systems Due to internal differences (e.g. buffer sizes)Due to internal differences (e.g. buffer sizes)

R&D will sometimes require very large-scale testbeds, R&D will sometimes require very large-scale testbeds, which could be provided by large farms foreseen for LHC which could be provided by large farms foreseen for LHC computing and Grid projects.computing and Grid projects.

If we desire to use commodity technologies, then If we desire to use commodity technologies, then anticipate use of 10 Gbit Ethernet, as well as Gigabit anticipate use of 10 Gbit Ethernet, as well as Gigabit Ethernet in SLHC systems.Ethernet in SLHC systems.


Data Sources / Readout BuffersData Sources / Readout Buffers This is electronics that buffers detector data at the input This is electronics that buffers detector data at the input

of HLT/DAQ systems and that sources data into of HLT/DAQ systems and that sources data into HLT/DAQ networks.HLT/DAQ networks.

They tend to operate at highest rates They tend to operate at highest rates (relative to other HLT/DAQ components).(relative to other HLT/DAQ components).• Cannot benefit from ‘event parallelism’, Cannot benefit from ‘event parallelism’,

as exploited by other componentsas exploited by other components Each data source must function at full FLT rate.Each data source must function at full FLT rate.

Increased occupancy (and increased FLT rate) increase Increased occupancy (and increased FLT rate) increase data source internal bandwidth requirements.data source internal bandwidth requirements.

These elements likely to need upgrade for SLHC rates.These elements likely to need upgrade for SLHC rates.• Compare performance with SLHC requirementsCompare performance with SLHC requirements


Data Sources / Readout Buffers R&DData Sources / Readout Buffers R&DUpgrade directions:Upgrade directions:

• Internally, these elements tend to employ busses for Internally, these elements tend to employ busses for data transfer.data transfer. Reasonable for O(Reasonable for O(80Mword/sec80Mword/sec) on circuit board) on circuit board Challenging for higher bandwidths between groups of Challenging for higher bandwidths between groups of

modules (modules (e.g.e.g. on backplanes) on backplanes)• High-speed serial connections may provide better High-speed serial connections may provide better

data transfer in upgrades.data transfer in upgrades. Technology trends in this directionTechnology trends in this direction

• FEE inputs to these elements already on serial links.FEE inputs to these elements already on serial links.• Serial output links to HLT/DAQ network with Serial output links to HLT/DAQ network with

bandwidth comparable to bandwidth of input links will bandwidth comparable to bandwidth of input links will remove bottleneck.remove bottleneck.

• I.e.I.e., push network technology closer to FEE, , push network technology closer to FEE, to exploit:to exploit: High-speed serial linksHigh-speed serial links Parallelism afforded by networkingParallelism afforded by networking


HLT AlgorithmsHLT AlgorithmsNew HLT algorithms neededNew HLT algorithms needed

To maintain selectivity in face of increased occupancy and pile-upTo maintain selectivity in face of increased occupancy and pile-upe.g. Electron triggerse.g. Electron triggers

• Dominate FLT output rate, Dominate FLT output rate, unless thresholds raised highunless thresholds raised high FLT rate = 22 kHz @ 30 GEV threshold at LHCFLT rate = 22 kHz @ 30 GEV threshold at LHC FLT selectivity degraded because pile-up blurs isolationFLT selectivity degraded because pile-up blurs isolation

• HLT algorithms must recovery selectivity despite degraded HLT algorithms must recovery selectivity despite degraded isolation.isolation.

How can this be accomplished? Will refined tracking information How can this be accomplished? Will refined tracking information need to brought to bear early in selection sequence?need to brought to bear early in selection sequence?

e.g. Muon triggerse.g. Muon triggers• Degraded by occupancyDegraded by occupancy• Degraded by loss of momentum-resolution at higher thresholdDegraded by loss of momentum-resolution at higher threshold• HLT algorithms must recover selectivity.HLT algorithms must recover selectivity.• HLT algorithms must accommodate upgraded muon detectors.HLT algorithms must accommodate upgraded muon detectors.

Moore’s Law will provide data processing power for new algorithms.Moore’s Law will provide data processing power for new algorithms.


Complexity HandlingComplexity HandlingHLT/DAQ systems are extremely complex.HLT/DAQ systems are extremely complex.

• Large numbers Large numbers (thousands)(thousands) of processors of processors• Even larger numbers of software processes & tasksEven larger numbers of software processes & tasks• Highly distributed, heterogeneous systemHighly distributed, heterogeneous system• ““Real-time” demands Real-time” demands not present in offline systemsnot present in offline systems

• Complex control Complex control (e.g. startup & shutdown)(e.g. startup & shutdown) procedures procedures• Remote access required Remote access required for monitoring & troubleshootingfor monitoring & troubleshooting

• Very high reliability requiredVery high reliability required Robustness, redundancy, fault toleranceRobustness, redundancy, fault tolerance Including robustness of complex selection algorithmsIncluding robustness of complex selection algorithms

Note: Technology evolution may mean that SLHC Note: Technology evolution may mean that SLHC HLT/DAQ systems are no larger than for LHC. In this HLT/DAQ systems are no larger than for LHC. In this case, SLHC ‘complexity’ stays same as LHCcase, SLHC ‘complexity’ stays same as LHC


Complexity Handling R&DComplexity Handling R&D

R&D can develop solutions to manage complexity.R&D can develop solutions to manage complexity.R&D can track development of tools for R&D can track development of tools for

‘complexity handling’ in very large commercial ‘complexity handling’ in very large commercial and other applications.and other applications.• E.g. web tools from e-commerce for high-level E.g. web tools from e-commerce for high-level

controls and user interfacescontrols and user interfaces Similar remote access, security, database issuesSimilar remote access, security, database issues


SummarySummaryHigher Luminosity =>Higher Luminosity =>

• Increased event pile-up & detector occupancyIncreased event pile-up & detector occupancy• Increased trigger rates & data volumeIncreased trigger rates & data volume• Increased radiation effectsIncreased radiation effects

Enabling technologies:Enabling technologies:• Integrated circuits: custom, FPGAs, memories, Integrated circuits: custom, FPGAs, memories, etc.etc.• Commodity computing & networkingCommodity computing & networking

Challenges arise in data transfer & in data processingChallenges arise in data transfer & in data processing• Processing challenges felt mainly in First Level TriggersProcessing challenges felt mainly in First Level Triggers• Data transfer challenges felt in both FLTs and HLT/DAQData transfer challenges felt in both FLTs and HLT/DAQ

Trigger rates manageable by:Trigger rates manageable by:• Increasing thresholds on inclusive triggersIncreasing thresholds on inclusive triggers• Using exclusive triggers where low thresholds neededUsing exclusive triggers where low thresholds needed

Technical solutions exist, but R&D is required.Technical solutions exist, but R&D is required.


R&D SummaryR&D SummaryFirst Level TriggersFirst Level Triggers

• Operation with 80 MHz crossing rateOperation with 80 MHz crossing rate• Coping with increased occupancy Coping with increased occupancy (muons)(muons) & pile-up & pile-up (calorimeter)(calorimeter)• Adapting to new sub-detectors or readoutAdapting to new sub-detectors or readout• Radiation-tolerant on-detector FLT electronicsRadiation-tolerant on-detector FLT electronics• Input data links Input data links from Front-end Electronicsfrom Front-end Electronics

High Level Triggers & Data AcquisitionHigh Level Triggers & Data Acquisition• Coping with increased data bandwidthCoping with increased data bandwidth

Input data links, HLT/DAQ networks, data sources / readout buffersInput data links, HLT/DAQ networks, data sources / readout buffers• Coping with increased occupancy & pile-up Coping with increased occupancy & pile-up (& new detectors or r/o)(& new detectors or r/o)

New HLT algorithmsNew HLT algorithms• Complexity handling Complexity handling

Date post:	11-Feb-2016
Category:	Documents
Upload:	lenora
View:	40 times
Download:	0 times

Andrew J. Lankford University of California, Irvine

Documents