Online monitoringfor ProtoDUNE-SPJingbo Wang, Antonino Sergi, Marco Adinolfi
ProtoDUNE-SPDAQCommissioningMeeting
January 8, 2018, CERN
Online monitoring
§ Duringtherunitwillbeessentialtomonitortheperformances ofthedetectorsandmake sure everythingisfunctioningasexpected.
§ Theonlinemonitoringrunswith the DAQ duringdatatakingandupdatesatregularintervals to indicateanypossibleproblems.
§ Themonitoringmodulesarewritteninartandwillunpack therawdataandproducethemonitoring plots during the run.
§ Atthemoment,thebasiconlinemonitoringframeworkhasbeenbuiltandisbeing tested.
Slide2
Candidate Monitoring Metrics
§ Generalmonitoring– Numberofsubdetectorswith data– Eventsizevs“time”– Triggerratevs“time”
§ Event monitoring– Raweventdisplayinthreeviews for each APA– Mean/RMSoftheADCsvs channel number– FFTofwaveforms vs channel number– Stuckbitsfraction– Trigger type
§ Plotswithbasicreconstruction (dependingonthecomputationalcapability)– NumberofhitsperviewintheTPC– Numberofhits/averageamplitudeperviewvstime– Distributionof clusterlengthandnumberofhitspercluster
§ Hardwarediagnosticinformation(forelectronicsexperts)– Seekingsuggestions
Slide3
Architecture of OM software
Slide4
§ DataLogger sendseventstooneormoredispatcherprocesses.
§ Dispatcherrouteseventstotheonlinemonitoringprocessesthroughsharedmemory.
§ Rawdecodersunpacktherawdatafragments(Timing,RCE,FELIX,SSP)usingtheoverlayclasses,andperformlow-levelanalysis.
§ Artanalyzersperformhigh-levelanalysisoftheunpackedevents(offlinecompatible)
§ HistogramsaresavedlocallyintheROOTfiles.
§ Webdisplaytool(Monet/LHCb)copiesthehistogramstothewebserver
Installation of the online monitor
§ Installation of the online monitor (JohnFreeman):– https://cdcvs.fnal.gov/redmine/projects/dunetpc/wiki/Guide_for_protoDUNE_online_monitoring_d
evelopers
§ Installation of the web-display tool Monet (Marco Adinolfi):– https://twiki.cern.ch/twiki/bin/view/CENF/PdspMonet
§ Theonlinemonitor is installedin the CERN computer np04-srv-014:/nfs/sw/om– Using a dunetpc feature branch “feature/online_monitoring_artdaq” (depends on both
lbne-raw-dataanddune-raw-data) with the raw decoders– Using the develop branch of dune_raw_data that provides the overlay classes and the
channel mapping service.
§ Note: the RCE overlay class depends on an external library that is installed in theProto-DUNE computer at CERN. At the moment, the online monitoring software isonly functional on the CERN machines. Fermilab (Tom Junk) is working to mergethis library to dunetpc for offline use.
Slide5
Running the online monitor
§ Basic operation: https://twiki.cern.ch/twiki/bin/view/CENF/PdspOMOp (to beupdated)
§ Run from the DAQ interface– The configuration files (fcl files) are stored in np04-srv-014:/nfs/sw/om/fcl– RawDecoder.fcl defines the parameters for all the decoding and analyzing modules– RunOnlineMonitor_ShMem.fcl is called by the DAQ interface to analyze the live data
during a run– StartOM_lite.sh starts a single OM process and saves the process ID to a txt file.– StopOM_lite.sh reads the process ID and kills the OM process– The output root file and log message file are saved to a local path: /scatch/OMoutput to
be accessed by Monet.
§ Run from the terminal– source/nfs/sw/om/ProtoDUNE_OM/setupONLINEMONITORING– Online with live data: art –c RunOnlineMonitor_ShMem.fcl– Offline with an existing file: art –c RunRawDecoder.fcl RawRootFile.root
§ Multi-process script was prepared and will be tested soon. Slide6
Example fcl file
Slide7
np04-srv-014:/nfs/sw/om/fcl/RunOnlineMonitorShMem.fcl
Unpack the rawdata and write tooffline format
Analyze the event withoffline format
Data transferred throughshared memory
Channel mappingservice
Online monitoring specificservice allows for file switchingduring the run (per 100 events)
Producers and analyzers are ranin sequence and no intermediatefile needs to be saved to the disk
RCE/FELIX Channel Map
Slide8
Onlinechannel
RCE,FELIXID
RCE,FELIXchannel Regulator
Regulatorpin
Asic Asicchannel APA
APAplane
Offlinechannel
§ Modified the 35-ton channelmap service and the map file
§ RCE/FELIX fragment ID andinternal channel ID are knownparameters obtained from theDAQ
§ Mapping the RCE/FELIXparameters to oneline/offlinechannel numbers
§ Regulator and asicinformation are not used atthis moment, but will beuseful for coherent noiseremoval
§ APA and plane columns maybe not needed because theycan be obtained from theoffline channel number withthe geometry service
RCE and FELIX Monitoring
§ Raw decoder (art producer module)– Number of ticks– All ADC values– Online and offline channel IDs– Expect more electronics diagnostic input: asic ID,
asic channel?§ Monitoring analyzers (art analyzer modules)
– Raw event display in three views– ADC Mean/RMS VS channel ID for each view in each
APA (2D histogram)– Profiled histogram of the ADC Mean/RMS– FFT spectrum VS channel ID (2D histogram)
Slide9
ü FELIX is assigned to the last APA.
ü In the analyzer module, rawdigits from RCE and FELIX aremerged to a single “TPC” vector
ü Analyzer modules work foroffline analysis with simulatedfile as well
ü Channelmapstillneedstobesettled
For more available offline modules, seehttps://twiki.cern.ch/twiki/bin/view/CENF/DUNEProtSPOM
Test with simulation file
Slide10
SSP Monitoring
Slide11
§ Raw decoder (art producer module)– AllADCvalues– Numberofpacketsperevent– Peak(frommetadata)spectrumperchannel– Area(frommetadata)spectrumperchannel– Area(frommetadata)spectrumperchannel– One waveformperchannelperfile– FFTperchannel– Triggertypehistogram(internal,external)– Persistentwaveformsovermanyevents
§ Monitoring analyzers (art analyzer modules)– ADC Mean/RMS VS channel ID(2Dhistogram)– Profiled histogram of the ADC Mean/RMS– More suggestions?
ü Channel map needs to be settledü High-levelanalysis(FFT)canbe
movedtotheanalyzer
Plots added by Paolo Franchini
Timing Board Monitoring
Slide12
§ Raw decoder (art producer modules)– Timestampofthetimingfragment– Timedifferencebetweentwoconsecutive
fragments– Triggertype
§ Monitoring analyzers (art analyzer modules)– Suggestion?
PlotsaddedbyPhilipRodrigues
Web Display Tool: Monet
Slide13
§ Weballowsforeasymonitoringfromeverywherewithoutconnectingtoremotemachines
§ Proto-DUNEisusinga modifiedversion of Monetthat wasoriginallywrittenbyLHCb.
§ Monetis written in python and hasseveralusercases. The PromptDQM case ishacked for Proto-DUNE, asthatistheonethatreadsROOTfiles (done by MarcoAdinolfi).
ü Folder/page/histogramstructure allows to organizeplots for differentcategoriesofusers:shifter,expert,specialstudies...
ü /nfs/sw/om/Monet/Dune_page_file defines the pages
ü /nfs/sw/om/Monet/Dune_histo_file defines which histogramto present for each page.
CanbeaccessedwithinCERNat:http://np04-srv-014:8123/prompt_dq/
Preliminary plots in Monet: TPC
Slide14
ADCMean/RMSvsChannel#
Preliminary plots in Monet: TPC
Slide15
ADCMean/RMSvsChannel#(profiled)
Preliminary plots in Monet: TPC
Slide16
FFTvsChannel#
Preliminary plots in Monet: SSP
Slide17
Persistentwaveform
Preliminary plots in Monet: SSP
Slide18
Eventnumber,ADCs,Npackets,frequency,Triggertype,FFT
Preliminary plots in Monet: Timing
Slide19
Multi-process running
Slide20
§ Themodulemustbeveryefficientsincethemonitoringneedstoallowsupervisionoftherun
§ Multi-process capability allows faster running and scalability to larger detectors§ Multi-process running was tested with the old set of monitoring modules. We’ll
update the scripts soon.§ Art doesn’t support multi-process within a module, so we chose to implement the
multi-process feature in the terminal– ./Makefclconfig.sh creates a number of fcl files to be processed– ./StartOM.sh launches all the monitoring processes and saves the PIDs.– ./StopOM.sh reads the PIDs and kill all the processes.– Lastly, the root files of different process have to be merged into a single file.
The Prescaler module allows to assign events to differentprocesses.
Things to do
§ Will clean up the existing code and add all the produced histograms toMonet.
§ Willkeepaddingmorehistograms astheyarerequestedorrequired.§ The channel maps for the RCEs/FELEX and SSPs have to be settled.§ Monet is slow, especially when presenting 2-D histograms. Need to select
appropriate bin size to increase the uploading speed. Bugs in Monet?(PhilipRodriguesfixedabugandmadeMonetfaster)
§ Need to update the multi-process scripts and add the feature to the DAQinterface
§ The shared memory plugin doesn’t allow to transfer data across nodes.Need to switch to multi-cast plugin (John Freeman)
§ The online monitoring modules can be directly used by offline analysiswith very little modification (Fermilab).
Slide21
Conclusion
§ We haveamonitoringframeworksetupforreadingtheraw dataandproducingdataqualityplots during the run.
§ We are developingand improving themodules. We’ll add more plots asthey are required.
§ The web display tool is under development and will be the next focus.
§ The online monitor is currently being tested alongside the DAQ. Potentialissues could be found. PleasereportbugsbyE-mailtotheonlinegroup.
§ Any suggestions are very welcome!
Slide22