The scope of this document is to describe a possible architecture for photo‐multiplier(PMT) readout, following the indication of a baseline structure agreed at the Paduameeting of October 2015.
=====================================================
1
Essentially, the baseline structure states that each PMT should embed both the High Voltage and the Readout electronics in a standalone manner, with sufficient I/O for PMT Signal processing, high voltage interface, triggering support and data readout.
The Juno collaboration recommends a basic structure in which PMT readout, trigger Primitive generation, fragment buffering, selective data readout and HV interface takesplace on a per PMT basis.
One possible solution is to embed these complex tasks on the PMT itself, by augmentingthe PMT physical volume with a water‐tight box housing HV and digital electronics and communicating with the external world by means of copper cables with an estimated length of ~ 100 meters.
Given the high (~20.000) number of PMTs involved, the overallcost of the digital board and the number of cables are to be assumed as parameters to be minimized.=====================================================
2
The conceptual scheme for trigger and readout electronics is summarized here.
Each PMT signal gets continuously digitized by a custom made ASIC and buffered in a Local memory while looking on the fly for meaningful data with threshold comparison:
in such a case a trigger request is generated and routed to a Global Trigger Processor via a dedicated link.
Data are stored waiting for a trigger accept that may come via an Ethernet interface:in such a case, a processing element will inspect the data storage looking for an event fragment (or a bunch of event fragments) matching the requested one(s) in a preset time window; upon positive match, the current timestamp and a centrally assigned event tag are associated with the matched fragment and sent to the Data Acquisition System through the Ethernet link.
This conceptual scheme is widely deployed in most modern physics experiments and is effective in reducing the bandwidth requirements of the data link. The worst case scenario in terms of bandwidth requirement comes from triggers due to dark current, whose rate may reach ~50KHz; assuming 30ns of sampling period at 1 Gsample/s with 16 bit words, the maximum data rate is in the order of 24 Mb/s, well in the
3
range of Fast Ethernet.
Hence the cost of the medium could be granted by an inexpensive CAT5e Ethernet cable.=====================================================
3
The problem of GCU synchronization in Juno is related to the distributed nature of datareadout.
Each GCU collects a fragment of an event which has to be joined with potentially many otherfragments of the same event before processing.
A barrel sorter approach through commercial telecom switches is a proven and efficient method to accomplish parallel event reconstruction in an event building farm, as schematically shown in figure.
Most physics experiments data acquisition systems adopt this technique because of scalability, excellentperformance at moderate costs, due to reuse of commodity widespread hardware. In Juno it makes even more sense if one adopts Ethernet readout at the very front‐end.
A Central Trigger Processor (CTP) collects trigger primitives generated at CGU level with fixedlatency and applies suitable algorithms (e.g. coincidence, multiplicity, windowing, etc.)according to physics constraints.
Upon reception of a suitable number of primitives that qualify a potentially interesting event, the CTP generates a tag related to a Global Time Counter (GTC) that must be shared among CTP, LCUs and GCUs. The time tag will then be forwarded to all GCUs via slow control or dedicated
4
link for reduced latency.
Upon reception of a time‐tagged event validation, GCUs will look for a match in theirstorage of buffered event fragments: the match may occur in a preset time window, e.g. inclose proximity of the time specified by the tag.
In case of a match, GCUs will retrieve the fragments from their memories and forward them to the data acquisition system (DAQ) via the IPbus dedicated link.
This readout mechanism features virtually no dead time but relies deeply on the availabilityof a global time at the level of GCUs. =====================================================
4
GCU’s will receive a global clock via a dedicated differential pair and they will count locally this clock thatwill act as a local copy of the Global Time Counter (GTC). But every GCU will experience an offset in theircopy of GTC. This offset has two contributions:
• the start of the counting is not synchronized among GCUs. Due to different powercycles and different time of arrival of reset commands in GCUs, their GTCs will start atdifferent times and will keep the same pace.
• The cables linking LCUs with GCUs are presumably of different lengths: the velocityfactor in cat 5 cables vary from 0.4c to 0.7c, e.g. a signal propagation speed of ~5ns/m.With an external clock of 62.5MHz, ~3m of length mismatch in clock cables aresufficient for a clock pulse sliding among GCUs
The figure shows the mechanism of global time synchronization between one LCU and itssubordinated GCUs and it can be summarized in the following list.
The synchronization procedure is preceded by a measure of round trip time (RTT) from LCUs and their GCUs in order to recover cable lengths mismatches.
1. DCS will put all GCUs in synchronization mode, by programming one of its registers
5
via IPBUS.
2. From that moment on, GCUs will ignore data and triggers coming from ADUs
3. GCUs will monitor their local copy of GTC, waiting for clock cycle in which their GTCwill match a preset value
4. At that exact time all GCUs will issue a technical trigger on the dedicated pair to theirLCUs
5. LCUs record the trigger time offsets among all their subordinated GCUs : thoseoffsets reflect the offsets of GCUs local GTCs.
6. Assuming that LCUs share a correct GTC, they can inform the DCS of the measuredcounter offsets in GCUs
7. DCS will adjust GTC in each GCU by adding the correct offset via IPBUS
8. DCS will instruct GCUs to issue another technical trigger when their GTCs match anew preset value in order to verify the correctness of the synchronization procedure
9. DCS will instruct GCUs to exit synchronization mode.
The synchronization procedure (except the RTTs measurements) can be repeated periodically(e.g. once per hour) in order to verify the consistence of global time. It is worth noting thatthe dead time generated by the procedure can be kept very low if the preset value at whichGCUs issue a trigger is close to the actual GTC.
The highest contribution to dead time will likely come from DCS operations via IPBUS, but weexpect that the full procedure may last in the order of hundreds of ms.=====================================================
5
According to figure the core components of the GCU are:
• ADU – a custom built ASIC with 1Gs/s ADCs, analog preamplifier and digital I/F forPMT signal processing; the digital I/F is LVDS double data rate (DDR) at 500MHz
• Storage : 2GByte high bandwidth, low cost RAM chips storing up to 1s of eventFragments
• Ethernet PHY: a 100Mb/s Ethernet physical layer chip for data readout and control
• PoE+ transformer : a 48V, 30W IEEE 802.3at compliant chip, receiving power supplythrough Ethernet cable in accordance with Alternative A
• A DC/DC converter feeding GCU with appropriate voltage rails
• Line driver and receiver: suitable buffers for clock reception and trigger transmissionvia ~100m copper cable pairs.
• FPGA: glue logic, high performance state machines, communication, ASIC readout,HV slow control
6
• Flash memory: long term storage for FPGA configuration bit‐stream; redundant fordisaster recovery=====================================================
6
The Ethernet medium requires protocols to be of any practical use, and Juno is no exception. In the context of the CMS collaboration, an effort has been made to develop and deploy an UDP based IP (Intellectual Property) core that abstracts an hardware bus on top ofthe Internet Protocol and is called “IPbus” . IPbus brings the UDP protocol at the FPGA level and leverages the ubiquity of IP for direct communication with distributed FPGAs through Ethernet. The IPbus ecosystem is comprised also of software APIs for a further layer of abstraction, that makes data acquisition and slow control for a high number of peer devices easier to manage. IPbus is deployed by several projects and experiments: CMS, ATLAS, LHCb, Compass, CBM, among others and neutrino experiments like SOLID and DUNE.
7
FPGA reprogramming is an important feature for bug fixing, feature enhancements, faultfixing and maintenance. Due to inaccessibility of GCUs after installation, the task is criticalbecause the firmware may get corrupted during the reprogramming phase itself. A safetymechanism must be put in place for disaster recovery.
The solution is to have the possibility to allow to choose between two firmwarerevisions (a production version and a “golden” version) stored in a flash memory. In presenceof external toggling, the FPGA would be presented and programmed with a productionfirmware at power‐up, while in absence of external toggle, the firmware selected forprogramming would be the “golden” version.
The toggle mechanism could be software triggered via IPbus or hardware supported, basedon the presence of an external signal, e.g. the externally supplied clock, as shown in fig. 6. Inthe latter case, when an external clock is not fed to GCU, the programming memory getsselected with the “golden” firmware, the FPGA runs on an internal clock and, after a powercycle, the FPGA will be configured with a known‐to‐be‐working firmware.
In case of bug fixes, firmware enhancements, tests, or GCU board qualification, the“production” firmware can be overwritten taking control of the flash memory data bus on theFPGA. With suitable support of IPbus mapped registers and FIFOs, the programming
8
procedure can be performed with simple software scripts.=====================================================
8
To reduce the number of copper pairs reaching the GCU, the IEEE 802.3at Alternative Astandard for remote power supply of (usually) telecom hardware may help. The standardforesees the injection of a ~48V bias on the central taps of the isolation transformers for thetransmission and reception pairs of 10/100 Mb/s Ethernet as shown in figure=====================================================
9
The remote powered device extracts the common mode voltage and feeds a local DC/DCconverter for appropriate use, sinking a maximum power of 30.5W.
Low cost commercial hardware is readily available to act as Ethernet Switch and PowerSourcing Equipment for as many as 48 ports
The same power source can be used to supply power to the High Voltage module, althoughsome effort has to be envisaged to shield the GCU from possible HV discharges.=====================================================
10
Here is shown a noise measurement that we have done on board that was developed for distributed control on accelerator.
We have implement a Power Over Ethernet Supply and we measure at derivate 1.2v that supply a microprocessor core byProbig with a 50ohm AC coupled probe manufactured by following the suggestion of a application note.
The result are good in half for our applications. We measure 7mV peak to peak and 700mV rms noise level=====================================================
11
In order to evaluate the performance of cat5e UTP cable coupled with cable drivers an equalizers we haveDone some measurements.
The test bench setup was composed by:
A Data a Timing generator (Tektronics DTG7274)
Evaluation boards of cable driver and receivers that are adapted for drive and receive signals coupled by a RJ45 connectors
100 meters of UTP cable and STP (shielded) cable for ferformance comparison
A advanced digital oscilloscope (Agilent DS691204A 12GHz bandwidth 40Gs/s ) for jitter ad data eye opening evaluation=====================================================
12
For our test we have choose the Cable Driver an Cable Equalizer the Texas instruments DS30BA101 and DS30EA101 respectively.They are very near our applications requirements (and the evaluation boards are easily and fast available).
This is only a kind of choice, there are no reason to evaluate similar products from others dealers =====================================================
13
The evaluations regards :
62.5MHz reference clock jitter degradation after 100mt of cat5e cable
125Mbit/s data eye aperture after 100mt of cat5e cable
65.5Msymbol/s (125Mbit/s manchester 1‐2 decoded) data from time over threshold trigger transmitted over 100mt of cat5e cable =====================================================
14
First :
62.5MHz reference clock jitter degradation after 100mt of cat5e cable =====================================================
15
Here are shown the jitter of the 62.5MHz clock transmitted via 100 meters of cable helped by cable driver and equalizer.
We have compare the performance of a UTP cat5e cable versus STP cat5e cable.
The results says that the shielded cable have the best performance (180ps versus 30ps peak to peak time over interval error)=====================================================
16
Second :
125Mbit/s data eye aperture after 100mt of cat5e cable =====================================================
17
Pseudo Random Bit Sequence (PRBS) patterns are predefine patterns of serial data that are generated for highspeed serial link performance test. We have choose the PRBS‐9 that mean 9 bit of data payload.
Here are shown the eye opening at 125Mbit/s transmission via 100 meters of cable helped by cable driver and equalizer.
We have compare the performance of a UTP cat5e cable versus STP cat5e cable.
The results says that the eye opening is very good , and the shielded cable have better performance.=====================================================
18
Third :
65.5Msymbol/s (125Mbit/s manchester 1‐2 decoded) data from time over threshold trigger transmitted over 100mt of cat5e cable =====================================================
19
The performance of the link on the PRBS pattern are in half for qualify the serial link. But we have want to play alsoWith a “real world” data applications.
In JUNO experiments actual idea of local trigger data transmission is to send to the global trigger system a pulseThat have a duration given by the time over threshold of the PMT signal, the pulse is tagged by the GTC content.The pulse have time resolution at least od one system clock cycle .
A similar pulse cannot be send via AC coupled serial link, so need a nrz coding. A good candidate could beA simple manchester 1‐2 code that have the consequence to double the data rate=====================================================
20
The chosen pattern simulate a time over threshold pulse encoding that have 10 pulses duration
Here are shown the eye opening at 125Mbit/s transmission via 100 meters of cable helped by cable driver and equalizer.
We have compare the performance of a UTP cat5e cable versus STP cat5e cable.The eye opening is better than PRBS test because this simple pattern do not stress so much theLink in terms of inter symbol interference.
The results says that the eye opening is very good , and the shielded cable have better performance.=====================================================
21
In the next days we plan to refine the evaluations by repeat the same measurements with scenario nearest asPossible to JUNO application, and test the clock and data errors check using a FPGA evaluation board. =====================================================
22
We conclude that the cat5e shielded cable coupled with cable driver and equalizer is a good candidate for JUNO environment, butWe need to perform other test in order to well define the restrictions and constraint in use.
By example looking at the Degradation in performance by transmit the Ethernet traffic in the same jacket=====================================================
23
In order to do performance test and for starting now with VHDL code development we plan to arrange a test benchArranged around a Xilinx KINTEX family evaluation board that is already present in our laboratory.
The evaluation board have installed the Ethernet physical layer, a DDR3 ram and two FMC port were we can attach aEvaluation board of commercial ADC and the cable driver and equalizers for trigger protocols test=====================================================
24
Here a block diagram of the kintex evaluation board=====================================================
25
Here (again) the block diagram of the GCU=====================================================
26
Here (again) the block diagram of the first prototype of GCU where instead of the GCUThere is a FMC interface that permits to install commercial ADCs and/or the firstPrototypes od ADU
=====================================================
27
=====================================================
28
