Post on 12-Dec-2016
transcript
Time Stamping Overview
Ian Lazarus
(STFC Daresbury)
Overview of talk:
• Introduction:– What is Time Stamping?
– Why do we do it?– How do we do it?
– Problems to consider.
– Examples of Time Stamping
• Future work
What is Time Stamping?
• Associate time label with data from a channel, a sub-system or a system.
• Fixed time relationship to moment when channel fired or event generated
• Permits the use of these time stamps to correlate data within detectors, between detectors, between systems.
• Simplest form is to correlate multiple free running systems (c.f. 1991 RDT experiments at Daresbury NSF)
• Note: Hardware triggers can be used too if necessary but many benefits of TS are lost if data is thrown away in this way.
Why do we do Time Stamping?
• Eliminates dead time (but not pileup)• Allows software triggering which hugely increases flexibility:
– Software can look back/forward in time (time=buffer size)
– Can take 100% of low rate in delayed coincidence with high rate(minimal dead time losses)
– Can correlate multiple systems without cross-triggering cables.
– Can easily do multiplicity triggers in systems with huge channelcounts
– Eliminate huge number of trigger connections (no problems from cables, ground/isolation)
– Overcome latency in pipelined processing (digital or analogue)
• Summary: Better data selection
How do we do Time Stamping?
• Need stable clock, lined up everywhere (adjust delay/phase)
• Need local TS logic in channel/sub-system/system
• Need a regular pulse to “timeframe” data (and lack of data)
• Need a powerful processing system and lots of memory
• Can be used with analogue or digital signal processing.
• Can be used alongside conventional hardware triggered readout, but with limited results.
Clk
Sync
Insert timeframe in data
and check local TS count now.
27 28 29 30 31 32 33 34TS
Count
Problems to consider when Time Stamping?
• Need timestamps in ASICs (c.f. N-XYTER and STFC CZT ASIC, CALICE MAPS, planned R3B Si ASIC. Others?)
• Latency- doesn’t matter until we make connections to ancillary detectors-cross-system triggers with analogue triggers
• Buffer edge effects (time/channels)
• Time alignment (setup) is critical – sub ns for AGATA PSA
– for simple coincidence must be better than half a clock across all systems
– for ToF must be few tens of ps (or calibrated)
• ToF not easy- TS is only as accurate as the clock after alignment and jitter errors.
– For ps ToF timing we still need to use TACs
Clk
Discrim
TS Register
TS
Count27 28 29 30 31 32 33 34
29
Discrim
Clk
TAC Start TAC Stop
Problems to consider when Time Stamping?
• Need timestamps in ASICs (c.f. N-XYTER and STFC CZT ASIC, CALICE MAPS, planned R3B Si ASIC. Others?)
• Latency- doesn’t matter until we make connections to ancillary detectors-cross-system triggers with analogue triggers
• Buffer edge effects (time/channels)
• Time alignment (setup) is critical – sub ns for AGATA PSA
– for simple coincidence must be better than half a clock across all systems
– for ToF must be few tens of ps (or calibrated)
• ToF not easy- TS is only as accurate as the clock after alignment and jitter errors.
– For ps ToF timing we still need to use TACs
– Should also calibrate the ToF channels for clock alignment errors
• Bandwidth for readout of high rate beam tracking (or else use links to feed in lists of interesting timestamp ranges to local readout)
Examples of NP Time Stamping Systems
• GREAT TDR (UK/Finland, Jyvaskala)- working
• CENTRUM (GANIL)- working• CENTRUM plus ATOM (GANIL) under design
• AGATA GTS system (Europe) designed; being built
• BUTIS (GSI/FAIR) Built (local distribution being specified)
• AIDA (DeSpec)• n-Xyter ASIC
Time Stamping example 1 -GREAT TDR
Rack includes
• 4 NIM and 2 CAMAC
for front end Amps, CFDs.
• A VME crate (data merger, metronome and
pattern unit)
• 2 VXI crates containing
between 12 and 15 32
channel cards (up to
480 channels).
Single stream of
time ordered data
to event builder.
Online
Analysis
Event
Builder
Tape
server
Tape
Time-ordered input data stream
Time of flight (5us)
start
Recoil data from focal plane
Ge data from target array
Time window
Coinc window (1us)
Selected data (6us)
GREAT data selection
Time Stamping example 2 -CENTRUM
OUT
GANIL
CLRC
TX 1-7RX
RX
TX2
TX1
RTAG LTAG1
TX5TX4
TX3
TX7TX6
LTAG2 LTAG3
LTAG4 LTAG5
LTAG7LTAG6
IN
CKEVT
IN
TEST2
DSP
IN
VXI-C
IN
IN
+ -ecl
TEST1
TST
SCLK
IN
AUX.
CENTRUM
•Time Stamps Events (not channels)
•Correlates
EXOGAM with VAMOS and TIARA
•ATOM upgrade adds channel timestamps
Time Stamping example 3 -AGATA
GTS ROOT
Ge
DIGITISER
PP
SEG
PP
Core
FADC FADC
DATADATA
Clk &
commands
Trig Req
“synch”clksynch
clksynch
Data Data
d15 = Synch or Inhibit
Test
Pulser
Test
pulse
Ge
GTS MEZZ
PP
SEG
PP
CoreDATA
DATA
Clk &
commands
Trig Req
“synch”clksynch
clksynch
Data Data
Test
pulse
PP core sends
special trig req
to GTS mezz
when it gets
d15 (10 or
20ns wide)
GTS MEZZ
MGT MGT
DIGITISER
FADC FADC
d15 = Synch or Inhibit
Test
Pulser
MGT MGT
GTS ROOT
Ge
DIGITISER
PP
SEG
PP
Core
FADC FADC
DATADATA
Clk &
commands
Trig Req
“synch”clksynch
clksynch
Data Data
d15 = Synch or Inhibit
Test
Pulser
Test
pulse
Ge
GTS MEZZ
PP
SEG
PP
CoreDATA
DATA
Clk &
commands
Trig Req
“synch”clksynch
clksynch
Data Data
Test
pulse
PP core sends
special trig req
to GTS mezz
when it gets
d15 (10 or
20ns wide)
GTS MEZZ
MGT MGT
DIGITISER
FADC FADC
d15 = Synch or Inhibit
Test
Pulser
MGT MGT
Alignment process: step 1
RocketIO
NODE MEZZANINE
RocketIO
TDC
Start Stop 1 Stop 2
TDC BOARD
ROOT MEZZANINE
Pulse
Generator
2
121
OSOSOSUO
TTTT ≅−=
UOT
The target is determining the uplink time (Tu) without RocketIO MGTs. In this case Tu is about the half of the loop time.
Alignment process: step 2
CHT
RocketIO
NODE MEZZANINE
RocketIO
TDC
Start Stop 1 Stop 2
TDC BOARD
ROOT MEZZANINE
Pulse Generator
2
111
OSSUOSCH
TTTTT −=−≅
In this case RocketIO Transceivers are enabled only for downlink channel.
Alignment process: step 3
FPGA
RocketIO Transceiver
RECCLK
DATA
DATA
RX FIFO
FILTER
TX FIFO
DELAY
FIFO
DLL
SFP
DATA
DRECCLK
FINE DELAY
(0÷2047 - 10 ps/step)
COARSE DELAY
(0÷255 - 10 ns/step)
USER CLOCK
USER DATA
UPSTREAM
DATA
TOWARD
ROOT NODE
Once the channels
latencies are known,
shorter latencies can be increased in each user
node. The result is that
each node will be
aligned to the longest
node latency.
Systematic Errors
L = 1 [m]
N 1 N 2 N 3
L = 1 [m]
N 1 N 2 N 3
Estimated Latency Error (without MGTs)
0,00
0,05
0,10
0,15
0,20
0,25
0,30
0,35
0,40
0,45
-1,7 -1,6 -1,6 -1,5 -1,4 -1,3 -1,2 -1,2 -1,1 -1,0 -0,9 -0,8 -0,7 -0,7 -0,6
Latency error [ns]
Relative Frequency
node 1
node 2
node 3
Node Latency (without MGTs)
0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
16,1 16,3 16,4 16,6 16,7 16,9 17,1 17,2 17,4 17,6 17,7 17,9 18,1 18,2 18,4 18,6
Latency [ns]
Relative Frequency
L1
L2
L3
EL1
EL2
EL3
Time Stamping example 4 –NUSTAR (BUTIS)
Si Ladder
DSSD
ASIC
FEE
Card
Handles
16 ASICs,
each 64ch
= 1024ch.
Total approx
100 FEE cardsBUTISBUTIS
Fan-out
NUSTAR
DABC
DAQR3B
input
code
module
Gbit Ethernet fibre(s)
ASIC
Fibre
Ethernet
R3B conceptual design
for Si tracker FEE and DAQ
Draft 1 April 2008
fibre
links
either fibre
or HDMI linksSlow
Control
Front End Electronics
Data output stage
standard format and
output medium e.g.
10G Ethernet fibre
Correlate by timestamp
Clock and
Timestamp
BUTIS
Common
Clocks
10/200MHz
<100ps/km
Slow Control
Common
database
loaded into
local
controllers
over Ethernet DetectorDetector
HV etc.
Time Stamping example 5 –AIDA (DeSpec)
Mechanics diagrams
Daughte
r card
connecto
r
16 channel
ASIC
AD9252FADC
AD9252
FADC
16 bit ADC
Sin
gle
todiffn
.
16
1
I2C controls
N x 2 discriminator outputs
Readout controls
Clocks
Reset
10 LVDS20 pins
10 LVDS
20 pins
4
Daughte
r card
connecto
r
16 channelASIC
AD9252
FADC
AD9252FADC
16 bit ADC
Sin
gle
todiffn
.
16
1
I2C controls
N x 2 discriminator outputs
Readout controls
ClocksReset
10 LVDS
20 pins
10 LVDS20 pins
4
Daughte
r card
connecto
r
16 channel
ASIC
AD9252
FADC
AD9252
FADC
16 bit ADC
Sin
gle
todiffn
.
16
1
I2C controls
N x 2 discriminator outputs
Readout controls
Clocks
Reset
10 LVDS
20 pins
10 LVDS
20 pins
4
Daughte
r card
connecto
r
16 channel
ASIC
AD9252
FADC
AD9252
FADC
16 bit ADC
Sin
gle
todiffn
.
16
1
I2C controls
N x 2 discriminator outputs
Readout controls
Clocks
Reset
10 LVDS
20 pins
10 LVDS
20 pins
4
Xilinx XC5VFX70T-?FF1136
640 I/O
344 used for 4 ASICS
ContainsPowerPC processor
Readout State Machine
64 channels of
Energy/Timing/Waveform
4 channels of ASIC readout
Timestamp logic
TEMACPHY
64MBSDRAM
FLASH forboot load of
LINUX
RJ45
Timestampexternalinterface
Consoleinterface
Oscillators
AIDA - FEE (64) - ASIC to DAQ block diagram
High speed Discr OR
JTAG
Gbit Ethernet
BuTiS interface
Water cooled metal
Water cooled metal
FPGA & FADCs
FPGA & FADCs
Water cooled metal
Water cooled metal
View A
Kapton Cable toto detector
RJ45Gigabit
RJ45Gigabit
HDMIClock,SYNC,ResetPower, Disc_out
HDMIClock,SYNC,Reset
Power
JTAG
JTAG
Connector board View A
One of 4
One of 64
Microblaze running
LINUX
Max rate = ( 60K x 8 )
+
( 10k x 2024 )
=>
20,720,000
bytes/sec
Include edge events
=>61,200,000
bytes/sec
SharedRAM
Max rate 60 K events/sec
Shared
RAM
Shared
RAM
Shared
RAM
Max rate 60 K events/sec
Max rate 60 K events/sec
Max rate 60 K events/sec
State
machine
Q
ADC
ASIC
16
channels
Timestamp
16 Discriminators
Sync/Pse/Res
To D
igita
l
Max rate 30K events/sec - 16 bit data
State
machine
Q
1024 x
(48 + 64 + 3)
Energy MWD
Leading Edge
discriminator
Slow in Fast out Waveform RAM
"Circulating Buffer"
Timestamp
Sync/Pse/Res
1024 long (20uS ) Floating pointtime vernier
calc.
Q
10 x 36Energy
DMA
Memory
manager
64MByte
DDR
SDRAM
32 bit
FIFO3072 x 16
VHDL block diagram
Patrick Coleman-Smith STFC, Daresbury Laboratory
Time Stamping example 6
– n-xyter
Time Stamping- future work in Hi/Despec
• House-training BUTIS?– Simple interface module– Keep down the costs per FEE card
• Connect AGATA DAQ– exchange timestamps
• Could we make a common TS port?– c.f. AIDA for DeSpec
• Common hardware interface, e.g. HDMI
• Make signals available to FPGA (change VHDL)