Muon dataflow
Edward Moyse
Introduction
• Turning the raw data / detector response into Nobel prizes, is a very complicated process, involving a lot of software
• … which of course means lots of potential bugs!
• In this talk, I will try to give an overview of the dataflow, so that you know roughly where to look in the case of problems
• I’ll also explain a little bit about some of the details of the sub-processes…
• Too much to cover everything!
• One thing to note: underlined words are links - this talk is supposed to be a starting point…
�2
DATAFLOW
Overview
�3
Bytestream
PRD
Roads
Segments MS Track SA Track
CB Track
ID Track
• Bytestream - raw bits and bytes from the detector readout
• RDO - Raw Data Object (object representation of the byte stream)
• SDO - Simulated Data Object
• PRD - Prepared Raw Data (after some calibration)
Sim Hits
RDO
SDO
Glossary
Digits
Simulation
(Usually) real data
Muon track Muon track extrapolated to the IP
DATAFLOW
Overview - links to (some) EDM packages
�4
Bytestream
PRD
Roads
Segments MS Track SA Track
CB Track
ID TrackSim Hits
RDO
SDO DigitsMuonDigitContainerMuonSimData
MuonSimEvent
MuonRDO
MuonPrepRawData
MuonSegment TrkTrack
https://svnweb.cern.ch/trac/atlasoff/browser/MuonSpectrometer/MuonDigitContainer/trunk/MuonDigitContainerhttps://svnweb.cern.ch/cern/wsvn/atlasoff/MuonSpectrometer/MuonSimData/trunk/MuonSimData/?#ada9ea6600c4c4e225be09a813a68677bhttps://svnweb.cern.ch/cern/wsvn/atlasoff/MuonSpectrometer/MuonSimEvent/trunk/MuonSimEvent/?#ac8b825360c824143e6e97bf4f4d64e1dhttps://svnweb.cern.ch/trac/atlasoff/browser/MuonSpectrometer/MuonRDO/trunkhttps://svnweb.cern.ch/trac/atlasoff/browser/MuonSpectrometer/MuonReconstruction/MuonRecEvent/MuonPrepRawData/trunkhttps://svnweb.cern.ch/trac/atlasoff/browser/MuonSpectrometer/MuonReconstruction/MuonRecEvent/MuonSegment/trunkhttps://svnweb.cern.ch/trac/atlasoff/browser/Tracking/TrkEvent/TrkTrack/trunk
DATAFLOW
Overview - links to (some) EDM packages
�5
Bytestream
PRD
Roads
Segments MS Track SA Track
CB Track
ID TrackSim Hits
RDO
SDO DigitsMuonDigitContainerMuonSimData
MuonSimEvent
MuonRDO
MuonPrepRawData
MuonSegment TrkTrack
https://svnweb.cern.ch/trac/atlasoff/browser/MuonSpectrometer/MuonDigitContainer/trunk/MuonDigitContainerhttps://svnweb.cern.ch/cern/wsvn/atlasoff/MuonSpectrometer/MuonSimData/trunk/MuonSimData/?#ada9ea6600c4c4e225be09a813a68677bhttps://svnweb.cern.ch/cern/wsvn/atlasoff/MuonSpectrometer/MuonSimEvent/trunk/MuonSimEvent/?#ac8b825360c824143e6e97bf4f4d64e1dhttps://svnweb.cern.ch/trac/atlasoff/browser/MuonSpectrometer/MuonRDO/trunkhttps://svnweb.cern.ch/trac/atlasoff/browser/MuonSpectrometer/MuonReconstruction/MuonRecEvent/MuonPrepRawData/trunkhttps://svnweb.cern.ch/trac/atlasoff/browser/MuonSpectrometer/MuonReconstruction/MuonRecEvent/MuonSegment/trunkhttps://svnweb.cern.ch/trac/atlasoff/browser/Tracking/TrkEvent/TrkTrack/trunk
Overview
�6
`
DATAFLOW
Bytestream
PRD
Roads
Segments MS Track SA Track
CB Track
ID TrackSim Hits
RDO
SDO Digits
RECONSTRUCTIONDIGITIZATION
BYTESTREAM DECODING
DATA PREP See Niel’s talk
Persistency
Configuration
• The main configuration (as is usual for Muon SW) is handled in MuonRecExample/MuonRec_jobOptions.py
• MuonRecFlags contains switches to turn off and on various parts of Reco/Dataflow.
!
• For users, parts are controlled in Reco_trf automatically by inputs and outputs
• i.e. doing Reco_trf inputRDOFile=myRDO.pool.root outputESDFile=myESD.pool.root will run normal muon reco
• From MuonRec_jobOptions.py, many other jobOpts files are called:
!
• For dataflow in particular, many live inside MuonCnvExample:
�7
https://svnweb.cern.ch/cern/wsvn/atlasoff/MuonSpectrometer/MuonReconstruction/MuonRecExample/trunk/share/MuonRec_jobOptions.pyhttp://atlas-computing.web.cern.ch/atlas-computing/links/nightlyDocDirectory/MuonRecExample/html/namespacepython_1_1MuonRecFlags.html#a8c13e425d4973b479d425591fb1815fhttps://svnweb.cern.ch/cern/wsvn/atlasoff/MuonSpectrometer/MuonReconstruction/MuonRecExample/trunk/share/MuonRec_jobOptions.pyhttps://svnweb.cern.ch/cern/wsvn/atlasoff/MuonSpectrometer/MuonCnv/MuonCnvExample/trunk/?#a54ce1fe31c8be632c9361454db65cb22
Digitisation
• Most packages are found under top level SVN directory MuonDigitization
• One (inexplicable?) exception: MuonDigitContainer, which is in the top level MuonSpectrometer directory.
• ‘Digitizers’ Algorithms (one per technology) call a IMuonDigitizationTool.
• Example: MDT_Digitizer calls MDT_DigitizationTool (see next slide)
!!
• It’s here that: wire sag, missing chambers, pileup etc etc are handled.
• Digits are stored in “IdentifiableContainers” - basically maps of collections of digits, where the key is the IdentifierHash of the collection
• Definition of IdentifierHash, can be found on MuonEventDataModel twiki, but for e.g. the CSC, the IdentifiableContainer has a collection (and IdentifierHash) per chamber - reading in a CSC collection, reads in an entire chamber’s worth of data.
�8
http://MuonDigitizationhttps://svnweb.cern.ch/trac/atlasoff/browser/MuonSpectrometer/MuonDigitContainerhttps://svnweb.cern.ch/cern/wsvn/atlasoff/MuonSpectrometer/MuonDigitization/MDT_Digitization/trunk/src/MDT_Digitizer.cxxhttps://twiki.cern.ch/twiki/bin/viewauth/AtlasComputing/MuonEventDataModel#IdentifierHash_table
MDT digitization
�9
MDT_DigitizerAlgorithm
MdtDigitizationTool
AlgTool, PileupTool
RandomNumberGenerators
MdtCalibrationDbSvc
IMDTConditionsSvc
• MDT_Digitizer calls an AlgTool (MdtDigitizationTool), which in turn calls a bunch of tools and services
• CalibrationSvc - which provides the t0 for MDT tubes
• ConditionsSvc - can provide a list of dead/masked chambers (also can be provided via jobOptions).
• IMDT_DigitizationTool - R/T relations
• MdtDigitizationTool reads in SimHits, and outputs Digits (MdtDigitContainer) and SDOs (MuonSimDataCollection)
• DigitsToRDO:
• Algorithms live in MuonByteStreamCnvTest(!) and have form, XXXDigitToMdtRDO.
IMDT_DigitizationTool
SDO!DigitsSimHits
MuonDetectorManager
DATAFLOW
https://svnweb.cern.ch/cern/wsvn/atlasoff/MuonSpectrometer/MuonCnv/MuonByteStreamCnvTest/?#a0a7378f95c480945c4457fdbdb91fcdf
Bytestream conversion
• Main package responsible is MuonByteStream
• Has several Algorithms responsible for conversations BS->RDO and RDO->BS
• BS->RDO: XXXRawDataProvider (XXX=Mdt/Csc/Rpc/Tgc)
• RDO->BS: XXXContByteStreamCnv (XXX=CscRdo/MdtCsm/RpcPad/TgcRdo)
• Both sets of algorithms call tools to do actual conversions:
• BS->RDO: Call tools implementing the IMuonRawDataProviderTool interface
• RDO->BS: Call tools implementing specific interfaces e.g. ICSC_RDOtoByteStreamTool
• Actual Tools are found in packages called MuonXXX_CnvTools (XXX=Mdt/Csc/Rpc/Tgc)
• These tools in turn call some extra tools - very technology dependent design at this point. Not much commonality
• One other important point - it’s here that the cabling services are called. These provide conversions from the online / offline identifiers.
• Found under MuonCablings - too detailed for this general overview talk
�10
https://svnweb.cern.ch/trac/atlasoff/browser/MuonSpectrometer/MuonCnv/MuonByteStream/trunkhttps://svnweb.cern.ch/trac/atlasoff/browser/MuonSpectrometer/MuonCnv/MuonCnvToolInterfaces/trunk/MuonCnvToolInterfaces/IMuonRawDataProviderTool.hhttps://svnweb.cern.ch/trac/atlasoff/browser/MuonSpectrometer/MuonCnv/MuonCnvToolInterfaces/trunk/MuonCnvToolInterfaces/ICSC_RDOtoByteStreamTool.hhttps://svnweb.cern.ch/trac/atlasoff/browser/MuonSpectrometer/MuonCablings
Muon RDO to Prep Data
• Algorithms are of form XXXRdoToPrepData, and found in MuonRdoToPrepData
• Again, most of the work is done using a convertor tool with interface IMuonRdoToPrepDataTool (in MuonCnvToolInterfaces) and concrete implementations in one of four packages, MuonXXX_CnvTools, and called XXXRdoToPrepDataTool
• e.g. MdtRdoToMdtPrepData calls MdtRdoToPrepDataTool, and which in turn calls a bunch of further tools:
• IMDT_RDO_Decoder, MdtIdHelper
• One final comment: the interface for the tool is designed for use offline AND in the trigger
• UNSEEDED MODE: Offline we decode retrieve the RDOs from Storegate and convert them all into PrepRawData
• SEEDED MODE: In the Trigger, the tool can DIRECTLY call the byte stream conversion for a passed collection of IdentifierHashes, i.e. we can decode the data within a ‘Region Of Interest’ (ROI)
�11
https://svnweb.cern.ch/cern/wsvn/atlasoff/MuonSpectrometer/MuonCnv/MuonRdoToPrepData/?#ac6e3b8ec7d85e96190d1689b8ddbf005https://svnweb.cern.ch/cern/wsvn/atlasoff/MuonSpectrometer/MuonCnv/MuonCnvToolInterfaces/trunk/MuonCnvToolInterfaces/IMuonRdoToPrepDataTool.h
Truth
• We already mentioned SimHits (MuonSimEvent), and SDOs (MuonSimData), but we also store some other objects which link the reconstructed objects to the truth (HepMCParticleLink)
• Won’t go into details here, but you can see them in MuonRecOutputItemList_jobOptions
!!!!!!!
• Also store Muon(Entry/Exit)Layer, which store the HepMCParticleLinks at entrance/exit of spectrometer
�12
https://svnweb.cern.ch/cern/wsvn/atlasoff/MuonSpectrometer/MuonSimEvent/trunk/MuonSimEvent/?#ac8b825360c824143e6e97bf4f4d64e1dhttps://svnweb.cern.ch/cern/wsvn/atlasoff/MuonSpectrometer/MuonSimData/trunk/MuonSimData/?#ada9ea6600c4c4e225be09a813a68677bhttps://svnweb.cern.ch/trac/atlasoff/browser/MuonSpectrometer/MuonReconstruction/MuonRecExample/tags/MuonRecExample-02-05-16/share/MuonRecOutputItemList_jobOptions.py
EDM/AOD Persistency
• Six (excluding truth convertors) main packages involved in Muon persistency
• Pool convertors (think of them as the interface to Athena/Storegate)
• MuonEventAthenaPool, TrkEventAthenaPool, TrackParticleAthenaPool
• Main documentation: https://twiki.cern.ch/twiki/bin/viewauth/AtlasComputing/WriteReadDataViaPool
• T/P convertors, called by Pool convertors and turn transient objects into something more ROOT-compatible, and handle schema evolution
• MuonEventTPCnv, TrkEventTPCnv, TrackParticleTPCnv
• Main documentation: https://twiki.cern.ch/twiki/bin/viewauth/AtlasComputing/TransientPersistentSeparation
• The list of objects written to ESD/AOD is defined in MuonRecExample/MuonRecOutputItemList_jobOptions.py
• Consists of lines like: MuonESDList+=["TrackCollection#MuonSpectrometerTracks"]
�13
Type Name
https://svnweb.cern.ch/cern/wsvn/atlasoff/MuonSpectrometer/MuonCnv/MuonEventAthenaPool/?#a1f668c6977af618c00894b8891ee01c2https://svnweb.cern.ch/cern/wsvn/atlasoff/Tracking/TrkEventCnv/TrkEventAthenaPool/?#af4d963e8ae61c9fd6ffa9c792e7d7690https://svnweb.cern.ch/cern/wsvn/atlasoff/Reconstruction/TrackParticleAthenaPool/?#ab68c5245d9e5720a9dc772d780ea5837https://svnweb.cern.ch/cern/wsvn/atlasoff/MuonSpectrometer/MuonCnv/MuonEventTPCnv/?#a58711ea705062b0258676b6a5bd44841https://svnweb.cern.ch/cern/wsvn/atlasoff/Tracking/TrkEventCnv/TrkEventTPCnv/?#a2af1c40766c57795eaa08d69c55e2ff6https://svnweb.cern.ch/cern/wsvn/atlasoff/Reconstruction/TrackParticleTPCnvhttps://twiki.cern.ch/twiki/bin/viewauth/AtlasComputing/TransientPersistentSeparationhttps://svnweb.cern.ch/cern/wsvn/atlasoff/MuonSpectrometer/MuonReconstruction/MuonRecExample/trunk/share/MuonRecOutputItemList_jobOptions.py
Conclusions
• The muon data flow is obviously very complex, and I can only give an overview here…
• Probably it could be simplified/cleaned up/documented better in some areas?
• Volunteers?
• But hopefully this gives an idea of where to look for problems
• … and of course, feel free to ask me, or on the muon sw mailing list, if you have further questions
�14