Pulsar System Overview and Commissioning Plan
Cheng-Ju LinFermilab
Installation Readiness Review02/08/2005
Outline: - Items from last review - Pulsar baseline system - Commissioning strategy - From now till March
From the last review, we were asked to provide a detailed list of action items that are essential for the commissioning. IRR committee expressed concerns about the long list of items.PC,TL2D,RC software issues:1.) Update trigger database to include new Pulsar software tags. (Daniel+database folks: Nov 12)2.) Test new trigger table built script. (Daniel+Kristian: Nov 12)3.) Generate TL2D in PC. (Kristian: Nov 1)4.) Porting command and control code to AMD machine. (Kristian: Oct 22)5.) Order two AMD machines. (JJ: target arrival date Nov 30)6.) Update/test FER code to readout Pulsar TL2D bank. (Cheng-Ju+Jane: Oct 29)7.) Measure readout time for TL2D and other configurations. (Cheng-Ju: Nov 15)8.) Update/test FER code to send scaler info to ScalerMon. (Cheng-Ju+Daniel+DAQ experts: Nov 19)9.) Update electron trigger code. (Kristian: Oct 25)10.) Update RC to switch between Pulsar and Alpha decision, confirm L2 tagset, L2 alpha auto reject. (Badgett, Tom Wright: 1st item is done, the rest complete by Nov 19)
Firmware Developments:1.) Isolation firmware improvement. (Sakari: testing ongoing, complete by Nov 5)2.) Implement firmware for additional system timing measurements and source ID. (Sakari: Nov 5)2.) ShowerMax firmware improvement and zero-suppressing. (Sakari: firmware testing ongoing)3.) SLINK package sending improvement to reduce transfer latency. The improvement is being tested on the SVT path first. (Sakari: testing in test-stand is done; Shawn+Sakari: ongoing testing in trigger room)4.) SLINK merger firmware improvement for timing optimization. (Sakari: Nov 10)5.) FREEZE all firmware development on Nov 15. Only bug fixes will be implemented.
ShowerMax:1.) Modify PulsarMon to accommodate new zero-suppression firmware and integrate into official PulsarMon. (Chris: by Oct 29)2.) Finish testing new firmware with ShowerMax "glitch" protection in test-stand and trigger room. (Chris+Sakari: Completed while I was making the list :-)3.) Checkout zero-suppression firmware in trigger room. (Sakari+Chris: Oct 29)
ShowerMax Active Splitter:1.) Test partial active splitter in L2 torture with low threshold; understand latent error rate. (Chris: Oct 22)2.) Modify AS to match light output of the splitter with the current system. Test modified boards (VLTN testing). (Chris: Oct 25-29)3.) If 1-2 are okay, install all AS cards/jumper fibers and test complete (splitted) system using patterns, low threshold runs in L2 torture. (Chris: Nov 5-15)
PulsarMon:1.) PulsarMon will be integrated into the official TrigMon release on Oct 25. (when Charles Plager returns from his west coast trip) (CP+Kaori: Oct 29)2.) Continue running a parallel TrigMon online (by Pulsar folks) to validate PulsarMon. (Wojtek+Vadim: ongoing)
Cluster Board:1.) Continue to work with Monica to resolve upstream protocol violation from CLIQUE board. The CLIQUE schematics were resurrected. Will study it with Monica.2.) Compile low threshold trigger into the new trigger tables (to try to re-produce cluster errors in the trigger room). (Vadim+JDL: Nov 1)3.) Continue to debug Iso-Cluster firmware: - Problem reproducible in L2 torture - New firmware available for testing (Sakari+Vadim+Wojtek: Nov 25)
Other Items:1.) Clean up cables in Pulsar and alpha crates. (Cheng-Ju+JDL: Oct 29)2.) The plan is to start re-integrate the Pulsar crate with all paths the week of Nov 15th.
ALL ITEMS ARE DONE!!!
SLINK
Pulsar pre-processors
L1 Muon
L1 TracksL1 Trigger decsion
TriggerSupervisor
L2 CAL(Cluster/ IsoCluster)L1 Et Sums
ShowerMax
SVT
Muon
Cluster
Electron
Merger
SVT
L2toTS
Baseline Level-2 Pulsar System
Merger
SLINK-PCI
Linux PCs:decision
PC
controlPC
RUNCONTROL
Supporting Infrastructure (unsung hero)
Pulsar/Alpha Trigger TableBuilding GUI
PC
Build Node
PC
Control Node
PC
Decision Node
ScalerMonitor GUI
RunControl GUI
Trigger Database
Run Summary
Level-3/CSL
Consumers(TrigMon,
PulsarMon,etc)
Pulsar CrateTL2DROL
TL2DTP2D
TL2DTP2D
SCALERSSCALERS
PRESCALECHANGES
HRR, CONFIG, ETC…
TRIGGERLIBRARIES
Personel In Charge• Pulsar firmware development: Sakari and Vadim• Control and Decision Node: Daniel and Kristian• Trigger algorithm development: Daniel• Trigger table building and database interface: Daniel• PulsarMon: Wojtek, Shawn, Chris, and Vadim• TL2D readout: Konstantin with support from DAQ group (Jane)• ScalerMon interface: Konstantin and Daniel• RunControl interface: support from DAQ group (Bill)• Fiber and LVDS splitting: Chris and Wojtek
Red = students
A very talented and hardworking group of people!!!
• In addition, we also get valuable support from the database and TDWG and DAQ groups.
Commissioning
Philosophy: do our best to minimize downtime to CDF physics data taking.
Two important items: - Split all input signal to the global Level-2 crates
(LVDS, HotLink and TAXI fibers). See Chris’ talk for ShowerMax fiber splitting.
- Have extensive validation and monitoring tools in place to ease the task of debugging the system PulsarMON.
Wojtek with the LVDS splitter board
Commissioning PhasesI. Purely parasitic running: - We’ve been running in this mode for the most part since Dec. 04. - The only operational impact is writing data to the event stream. - Able to debug essentially all firmware and trigger algorithm bugs in this mode. - Also useful in the showerMax fiber splitting effort.
II. Semi-parasitic (“symbiotic”) running: - Allowing Pulsar to handshake with Trigger Supervisor in physics runs. However, always send L2Reject. - Pulsar running in “operational-like” environment - We have been running in this mode fairly smoothly since Feb 2. Only a few operational glitches were discovered along the way. - Ran through > billions of L1A.
Commissioning Phases
III. Beam run: - So far we have done 2 EOS beam run with Pulsar. The second beam run was with Pulsar using full physics trigger table without silicon in the run (See Vadim’s and Daniel’s talk for details). - We able to run in various configurations (select via RC): * Alpha driving, Pulsar auto-reject * Pulsar driving, Alpha auto-reject * Pulsar driving alone. - Tested new TL2D readout code and ScalerMon interface * Pulsar TL2D is by design identical to RunIIa TL2D to ensure backward compatibility * When both Alpha and Pulsar are in the readout, two TL2D blocks are generated. The system does the actual driving will write to first block of TL2D
PULSAR TL2D Validation (beam test)
• Over 50K Pulsar TL2D events passed L3 without reformatter errors.
• Consumer monitors + CSL processed Pulsar TL2D without problems.
• Pulsar TL2D + TP2D (light-weight) readout time is ~550usec.
• Since Pulsar TL2D is identical to RUNIIa TL2D, we could actually use the RunIIa TrigMON TL2D monitor. In fact, we caught a problem with XTRP EOE word using the IIa Monitor.
• Will need a lot more events to fully validate the new readout code and TL2D bank.
• We would like to declare this readout code current soon. We are waiting for the offline validation to give us a green light.
PULSAR ScalerMon ALPHA ScalerMon
• Extremely useful to have two ScalerMon side-by-side for rate comparison (See Vadim’s talk for detail).
From Now Till March
• Will repeat EOS tests with silicon included during the next store. Have already extensively tested with silicon in COSMICS.
• We will request a few test runs with Pulsar driving at different luminosities.
• Continue to validate the new TL2D readout. Want to discover problem now rather than after 2 months of running.
• Continue to run in symbiotic mode. It’s already default in AAA_CURRENT and AAA_NOSILICON.
• Continue to improve control/decision node handling: eg. HRR and error handling.
More Items to Do Before March
• We have already established an ACE help page (thanks to super scico Young-Kee). As we learn more about our system in operation, we’ll update the page accordingly.
• PulsarMon has been used quite extensively in the commissioning. We are working on a CO friendly version to run as default in the Trigger room. We’ll update the web CO checklist accordingly.
• For commissioning, PulsarMon will be updated to taken advantage of the dual TL2D blocks to validate trigger bits and the rest of TL2D between alpha and Pulsar on the fly.
• We have sufficient number of spare Pulsar boards. We will need to put together a complete crate and set aside as hot spares.
More Items to Do Before March
• Will need to “officially” rebuild all the trigger tables using the new GUI so the linux table valid flags are set in the database.
• To the committee: anything else that I’ve missed?
Some questions from the committee: Q: What is the pager rotation plan?
We’ll follow the current system and rotate on a weekly basis. Initially, we’ll also have a local backup pager carrier. The duty of the pager carrier is limited to the Pulsar system. However, historically L2 pager carrier also responds to L2 CAL trigger problems (since Monica is offsite).
Some questions from the committee: Q: For how long do we need to maintain the legacy system?
We would prefer that the legacy system be maintained as hot spare (powered with full support) till at least the summer shutdown. Afterward, it could be maintained as “cold spare”.
Q: What hardware configuration will be used for operations. In particular, what signals will be split/fanned-out? We would argue to maintain the full signal split/fannout infrastructure until the legacy system becomes cold spare. Depending on the R&D need, we may want to support the splitter for an even longer term.
Some questions from the committee: Q: What is your VME crate/test-stand need?
B0l2pu00 will be the primary crate for operation. We’ll need at least another crate in the trigger room for L2 and XFT R&D. Right now we have 2 test-stand crates in B0. We would like to maintain those two crates for R&D work in the immediate future.
Plan/Request for March
• We would like to run for a week during the first week of March with the new TL2D readout code and trigger symbiotic mode (Pulsar auto-reject)
• We would then like to run for two weeks with Pulsar driving and alpha in auto-reject mode.
SUMMARY
• The commissioning effort is going extremely well. We have achieved all our major milestones.
• The remaining talks will provide the committee with details of the Pulsar status and test results to help the committee determine whether or not the Pulsar system is ready for operation.
• There are clearly still some loose ends that need to be followed through. We have a group of motivated young-blood to tackle the remaining issues.
