Tevatron IPM status Oct 2009 Andreas Jansson K.Bowie, T.Fitzpatrick, R.Kwarciany, C.Lundberg,...

transcript

Tevatron IPM status Tevatron IPM status

Oct 2009Oct 2009

Andreas JanssonK.Bowie, T.Fitzpatrick, R.Kwarciany,

C.Lundberg, D.Slimmer, L.Valerio, J.Zagel

Also thanks to:

T.Anderson, M.Bowden, A.Bross, A. Chen, R.Dysert, S.McCormick, S. Suleimani, H.Nguyen, C.Rivetta, H.Glass, D.Harding, B.Hively, V. Kashikin, D.Miller, Z.Tang, J.Volk, T.Zimmermann, …

Outline

• Brief overview history of Tev IPMs– Summary of Review talk from 2007 – Long version available at:

http://indico.fnal.gov/conferenceDisplay.py?confId=1272

• What happened then?– Progress reports from 2007 to now– Summary of partial reports given at Tev

Dept & Vaia’s meetings

• Current status

10/15/2009 IPM status October 2009

DESIGNBrief overview of IPM history

9/21/07 IPM Review, September 21, 2007 A. Jansson 4

GOAL:Measure protons and pbar beam size

turn by turn at injection and ramp during normal operation to diagnose

and mitigate emittance blow-up.

Challenges in the Tevatron

• Two small beams separated by helix.

• Separate protons from pbars, injected from circulating beam

• Beam induced parasitic signals.

• Low vacuum pressure

• Fine granularity and many channels

• Single bunch resolution and gating

• Improved shielding and matched cables

• Local pressure bump with controlled leak

Solutions:Challenge:

IPM detector

MCP1kV100V

9.9kV10kV

All signal cables are enclosed in a Faraday cage!

Anode board and internal cabling

• ¼ mm strip pitch• 200 channels (128

instrumented)• Board mounted series

resistor for back-termination and LP filtering.

• In-vacuum signal cabling using UHV-compatible flex-circuits

• High resolution area can be moved by swapping connectors

Microchannel plate

• Max gain with 36 proton bunches is ~1e4 to avoid saturation.

• Can be achieved with single plate

• With dual plates, each plate would run at a very low gain and low bias current.

• Use single MCP with extra-high bias current.

Front end cabling

• 40 channels in each vacuum DB50 feed-thru (flex circuit)

• Transition board from feed-thru.

• Mini-coax bundles with external shield.

• Custom (CMS) DB coax connector in FE board.

DAQ system

• CMS-QIE chip digitizes signal in tunnel.

• Serial data uplink on optical fiber.

• Receiver and data buffer in upstairs PC

• Timing + QIE clock + QIE clock supplied from PC thru cat-5E cable

Timing card(PCI)

Timingfanout

QIE cards(16x 8 ch)

Data Buffer(2x 8 links)

Host PC (LabView)

Service buildingTunnel

QIE card

• 8 channels (CMS QIE8) per board.

• Achieved noise ~1.8fC with 4’ cable.

• Data is combined with timing information, serialized by CERN GOL ASIC (rad hard) and sent thru optical fiber at 1.1Gbps data rate

• Timing fanout board cleans up and distributes clock and timing signals

FE board input/output format

7 6 5 4 3 2 1 0

15 14 13 12 11 10 9 823 22 21 20 19 18 17 1631 30 29 28 27 26 25 24

QIE1QIE2QIE3

QIE5QIE6QIE7

QIE4CAPID0Timing bits

TX_EN/RX_DV

QIE0QIE1QIE2QIE3

QIE5QIE6QIE7

QIE0QIE1QIE2QIE3

QIE5QIE6QIE7

Proton Revolution CounterError bits

QIE mode

2RF/7 Clock Counter

11000 No effect11001 Proton Injection Marker11010 Antiproton Injection Marker11011 QIE Mode0 (QIE in Calibration Mode)11100 QIE Mode1 (QIE in Mode)11101 QIE Mode2 (Link test - outputs a 32-bit counter to GOL)11110 QIE Mode3 (Reset QIEs)11111 Board Reset

• 15MHz clock frequency• Proton and pbar revolution markers•Encoded signal:

INPUT (Cat5 cable)

OUTPUT (1.6Gbps serial link)

Data buffer card

• Handles 8 incoming optical links (64 channels, 1.1 GB/s of data)

• Can sparsify data on–the-fly based on timing masks

• 512MB RAM allows for– 20.000 turns of continuous data– 90.000 turns for 72 bunches– 6 million turns for a single

• Read out thru PCI64 bus.• Two boards are used to

handle 128 channels.• Sync via jumper cable

• IPM buffer board doubled as prototype for BTeV L1 data buffer.

• Considered for use in MICE experiment.

Timing card

• Produces the 15MHz (2/7 RF) QIE clock

• Decodes and transmits beamsync revolution marker + injection and trigger events

• Controls QIE settings.

396 ns (21 buckets)

p-pbar separation ~120ns

Labview Software

Installation timeline

• Magnets installed shutdown ‘04• Vertical detector installed Dec’05

– Arcing problems inhibited operation at full B field. High outgassing from retrofit retainer.

• Vertical detector fixed, horizontal installed May’06– reduced DAQ system with single buffer board

due to power supply limits and firmware problems).

• Full DAQ, larger leak installed Oct07– 80 channels per plane, 2 buffer boards

System Status 2007

• Both IPMs had ‘new’ software and firmware.

• Vertical IPM works pretty well, basicall since installtion.

• Horizontal IPM has some hardware problems (bad FE baord) that required an access.

• Horz IPM also requires two buffer boards to measure beam size at injection

NB. REAL DATA!

Beam size at IPM location

By moving detector, able to fit beam in active area, except horizontal beam size at 150GeV

EARLY MEASUREMENTSBrief overview of IPM history

Protons at low beta

• Proton bunch #1 at low beta during store #4758.

• Measured beam size 0.55mm, turn-by-turn variation (noise) 20µm.

• Total signal per bunch ~1.3pC.

30 turn average

single turn

Magnet at 200A

Injection – Full B field

0.2 0.4 0.6 0.8

Proton bunch #21 turn-by-turn

Store #4772 Magnet at 200A

RMS profile width

0.0 0.5 1.0

0 10 20

T0 10 20 30 40

100 turns

Four injection measurements superimposed

Measured mismatch vectors

• Nominal settings• Q701 at -22A• Q701 at -42A• Q711 at -25A

• Need to increase Q711 by ~15A (saturation?)

• Should result in 5-10% smaller vertical emittance

• Should have very little effect on horizontal plane.

• Should have been tested this morning…

-0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8

Flying wire comparison

1 4 7 10 13 16 19 22 25 28 31 34

Flying wire

y = 1.0089x

0 0.5 1

Flying wire

Known electronics problem (cross talk from revolution marker pulse)

Comparison of vertical beam size from IPM and nearby Flying Wire.Tuning of abort gap cleaner timing had caused blow-up of certain bunches. From MAD lattice file, expect a 13% wider beam at Flying Wire. See ~1%.

Pbars at 980 Gev

Store #5172, Jan 07Horizontal Vertical

Pbars at 150 GeV

• Also saw coasting pbars (with vertical system) at injection

• Requires moving the detector to pbar orbit.

• Triggering did not work at first attempt (timing event found to be disabled).

Fri Jan 12 2007, shot #5175

Shot 5186 – vertical plane

Two consecutive beam revolutions in Tevatron, just at injection of batch 9.

Vertical pbar injection data (batch 9)

• Beam position gives consistent oscillation ~0.2mm.• Need to look closer at beam size data, but…• There is a clear frequency component at twice the

0 2 4 6 8 10turn

ezismm

Beam size [mm] Beam position [mm]

100 100

ISSUESBrief overview of IPM history

Parasitic signals

• Despite very good shielding, beam related signals still present with MCP gain off.

• Very stable from turn to turn -> reference subtraction possible

• Present even on “broken” channels (where no ionization electrons are seen)

0 50 100 150 200 250 300Turn #

langiSthgnerts

1x1 store at 980GeV ~center anode strip

Timing markersBeam pulse

Buffer board sync

• With two boards, capacitive background appeared unaligned, and background subtraction didn’t work well– Found and fixed

software data alignment bug

– Verified that buffer board sync check works properly

PCI card poll glitching SERDES

• Polling card via PCI during measurement was found to glitch the SERDES reciever.

• “Solved” by using interrupt instead of polling, and reorganising code to minimize PCI traffic during acquisition.

INTEGRATIONWhat happened more recently?

IPM Review 2007

IPM Review Committee ReportThe charge to the committee has consists of the following two items: • 1. What is the current status of the Tevatron IPM systems - the front-end electronics

and DAQ in particular? Identify any known problems or issues.• 2. Can the IPM system be made fully operational - complete detector readout with

production electronics and DAQ issues resolved - by Jan 1, 2008? What additional resources ($$, effort) would be needed to meet that date?

The committee was positively impressed with quality of the work and the part of the presentation devoted to the item 1. Nevertheless very little was said about the second half of the charge. It is unclear:

• What does it mean “to make IPM system fully operational?”• Which additional resources, if any, are required to finish the project timely and

which additional expenses are required?Taking the above into account the Committee suggests to convene an additional short

session devoted to discussion of item 2 of the charge. Andreas should settle with the Tevatron department what they consider for the project to be operational and to present a clear answer to the part 2 of the charge. Having this session in about one or two weeks should leave enough time to elaborate details.

Tev dept wish list

• In General the IPM controls and data retrieval should be very similar to the flying wire front end. It should be able to perform the following:

• – Decode V:CLDRST state and setup the instrument based on a state to instrument spec mapping owned by the

IPM front end. The TFW instrument has an example of this.– Should provide plane and particle sigma data on event $75 (profile timer) There are 35 profiles that currently

go out up the ramp and through the lowbeta squeeze.– Should provide data to SDA. The instrument should own a “data ready” device similar to TFW that

communicates the state of pending data. This can also be used for lumber jack Client Logging.– The instrument should be free running providing data during a collider store at ~ 1/30 min. (similar to TFW).– The instrument should write bunch by bunch data into ACNET array devices following the standard convention.

• . The ACNET array devices that hold bunch by bunch information should be 37 elements long. • The enumeration of array devices should have the [0] element of the array as the average value of the

36 array members to follow. There may have to be some special code to handle updating the average to reflect and average of elements with non zero values.

• The [1 – 36] would store the data where bunch 1 would correlate to [1] of the ACNET device respectively.•

– ACNET parameter representing data and setup should be placed in the lumberjack utilizing Client Logging.– The IPM should have an ACNET application that can perform the following functions (similar to T46 for TFW):

• Standard user interface.• View and Configuration of State to IPM Spec Transition Map for pre-loading specs.• View and Configuration of IPM specs.• Ability to access and plot profile data correlated to data taken on profile event and periodic sampling with

capability to have fitting options for plots.– IPM ACNET devices should attempt to follow a naming convention.

Tevatron IPM Specs for Collider II OperationGeneral Collider II IPM Specifications:

ACNET control

• Since early 2008 the Tevatron IPMs are state driven in much the same way as flying wires.

• Selected output is logged when data ready since mid-January 2008

9/3/2008 IPM status A. Jansson 36

Tom’s ACNET interface

Available summer 2008 (no plotting)

PSPEC settings

Pbar jacking

Jan 31, 2008Store #3874

Pbar jacking 2

Feb 1, 2008Store #3876

IPM–FW comparison

Jan 2008

How to improve data quality?

• Fluctuations from measurement to measurement much bigger than turn-by-turn fluctuations for coasting beam.

• One known source of bad data: synchronization problems!

Example: Synchronized

Example: Unsynchronized

Easy to detect and reject by eye for manual data-taking!

One manifestation of sync issues

50 100 150 200 250 3000

P rot on C ounter

Histogram of proton counter in first buffer data frame. Expect single cluster of three bins around zero.Saw wider cluster + scatter + unexpected peaks

Sync saga

• We discovered relatively early on that PCI access during an acquisition would sometimes glitch one or more of the high-speed data links.– Appears to be a FPGA internal problem, now

believe it is caused by does not like 5V input signals on pins close to high speed inputs.

– Work-around: re-order setup sequence to limit access to board, use interrupt instead of polling.

– Seemed to work fine.

Board layout

Sync saga cont’d

• Even without PCI access during acquisitions, loss of synchronization still occurred relatively frequently, particularly for injection measurements (long wait time)– Improved the initial synchronization logic– Added firmware logic to allow links to

resynchronize dynamically if needed, while waiting for a trigger (In original implementation, the entire chain was synchronized once by a timing card reset just before an acquisition)

Sync saga cont’d

• Early resync logic worked in test stand, but not in the Tevatron systems.• Used simulated TVBS in lab5 teststand.

• At this point, FY08 omnibus budget came out, furloughs started and we lost Kwame (our FE board engineer) to Florida.

• Rick (Buffer board engineer) took over debugging of both boards, and we moved the test stand to FCC (closer and real TVBS available).– Real TVBS available

Sync saga cont’d

• Found occasionally missing AA markers, and that FE board and buffer board treated this unspecified case differently.– Fixed with buffer firmware modification.

• Also discovered “unexplained” PCI accesses during acquisition?– Don’t know where they come from, or how

long they have been there (system patch?). – Tried to have board ignore PCI requests

during acquisition, but this caused total PC lock-up.

Status ~sept ‘08

• Improved re-sync logic got residual error rate down to per mil level (in test stand).

• Couple of “full firmware versions” tested OK in test stand but crashed the horizontal IPM host PC.

• Moved horizontal system to FCC for easier debugging.– Source of problem related to new inter-board

sync logic– Found bad power supply in horizontal system

• Swapped test stand PC with Horz system

Summary of sync issues

• Spontaneous front end resets (early on)– Revert to default mode and loss of synchronization.– Due to Single Event Upsets (or just noise on slow control channel?)– Fixed with new front end firmware version plus Labview software changes

• Trigger/Counter synchronization (affects both systems)– Links synchronized, but bunch data not in the correct time-bins.– Due to “missing” AA markers– Fixed by having both boards treat missing markers in same way.

• Intra-board synchronization (affects both systems)– Loss of SERDES lock or illegal character on serial link. – Due to “unexplained” PCI accesses– Effect minimized by re-sync logic. Errors during acquisition flagged.

• Inter-board synchronization (affects mainly horizontal)– Data from two buffer boards not aligned (varying from measurement to

measurement) affecting bunch profiles and background subtraction.– Was hard to debug in the presence of other sync issues…– Fixed with a combination of firmware and software changes

LabView software update

• Changed software to not issue “data ready” when a acquisition problem is detected– Only data without hardware errors

datalogged.

• Fixed ”total signal charge” readback, and chi-square readback.

Nov’08

Observations with “new” firmware

• The beam related background, which used to be significant but very stable, is now fluctuating a lot in horizontal plane.– Likely due to some sort of timing jitter

• Also, the signals appear to be smaller than before– Possibly due to new x-collimator ion

Beam background

0 5 0 1 0 0 1 5 0 2 0 0 2 5 0 3 0 0

“Average beam background”

“Fluctuation in beam background”

VIPM and HIMP both looking at the vertical signal

Various failed attempts to fix jitter

• Swapped timing cards• Changed timing card slot location

(possible interference with buffer board)

Status – Jan’09

• Observed jitter in horz clock/timing signals– Not reproducible in test stand– Swapping timing boards back and forth

gave inconclusive results.– Possibly related to fluctuating signal?

• In the meantime found and fixed bug in background subtraction for pbars.– Pbar data much improved

Jan’09

February power failure

• After the power outage on February 11, IPMs stopped reporting data to ACNET– Software detected persistent buffer errors bits, marking data as

bad• VIPM: Buffer 1/2 mismatch• HIPM: Pbar marker error

• Swapped cables, to locate error– Buffer mismatch followed upstairs PC– Pbar error followed downstairs crate.

• Pulled buffer boards from vertical system for bench test.– Left horizontal PC monitoring vertical signals

• Bench tested boards, finding nothing out of the ordinary– Upgraded test stand labview code and re-tested, saw nothing

• Reinstalled, buffer mismatch problem gone– Had to distable signals from horizontal front end board 1, which was

causing the pbar marker error.

• Mechanics now seems to be working properly

Signal strength issue

• Signals are smaller than they used to be, fluctuating a lot and occasionally disappear altogether– Not due to stochastics of gas

interaction– HV rise time ruled out as a culprit(?)– Gas pressure fluctuating?– Not currently understood

April’09

Fluctuating data

OK data

Marginal data

Bad data

Before shutdown ’09

• Tried turning off some ion pumps– Signals strength improved somewhat– Signal fluctuations still there

• Took vertical tbt data at injection– No obvious mismatch

• Found almost 1V ripple on 3.3V supply to HIPM timing card– Replaced power supply– Timing jitter now essentially gone

Signals just before shutdown

HIPM at injection

Improved error detection

During shutdown, improved the buffer error Detection to reduce false positives

Error rates in machine (without beam) consistent with test standwith few exceptions (weak boards). Swapped two boards.

STATUSCurrent

Data availability

• While we have been working on improving the measurement reproducibility, data from ramp, squeeze and store have been available in ACNET and logged since beginning of 2008.

• Injection and “pbar jacking” data taking have been enabled occasionally– Not enabled permanently due to concerns

about depleting MCP if stuck with high voltage on for long times.

Hardware, firmware and software

• On-the-fly re-sync logic seems to work

• Intra-board sync seems to work• Improved the error detection,

minimizing false positives• Small residual error rate (due to

SERDES glitches) remain. Any flawed data is discarded

Signal Quality

• Signal is weaker than before, in particular for horizontal system– Horizontal pbar signal no longer visible– Leak no longer as effective?

• Signal level is fluctuating significantly, causing fluctuation in measured beam sigma.– Gas puffs from pumps?

• Noise level also seem somewhat higher, at least in some channels

E0 gas pressure

Last 4 years

TSPs were fired during the shutdown

Degassing TSP filaments at IPMs (spoiling the TSP)Helped bring the signal back up somewhat

TSP degassing test

Trying to raise local pressure by degassing TSP filaments gavelittle or no effect on the measured signal (might have worked better if pumps were off)

Horz signal then and now

• 2007, before shutdown and now

IPM vs FW

9/3/2008 IPM status

Store #7243Oct 10, 2009

Consistent to 10% with FW, but weak IPM signals

Turn-by-turn data

Enabled injection measurement for store #7265

Got data!No apparent vertical mismatch (horizontal signal too weak)Some trigger issues (only a few bunches captured, rest of data stale)To be looked into further.

Summary

• Tev IPMs are delivering data since almost two years

• Compares quite well to FW, but some fluctuations in the sigma readings.

• All known hardware issues have been resolved, data affected by any remaining occasional hardware glitches discarded.

• Remaining issues related to weaker signal with larger fluctuations than before.

• Also seems there are some triggering issues for injection measurement (not used regularly)

Tevatron IPM status Oct 2009 Andreas Jansson K.Bowie, T.Fitzpatrick, R.Kwarciany, C.Lundberg,...

Documents