“PCPC Latency measurements”
G.Lamanna, R.Fantechi & J.Kroon(CERN)
TDAQ WG – 8.9.2011
Introduction Why the latency measurement is important
for us?: Primitives transmission to the L0TP through ethernet (Low level protocol, possibility to have switches, small spread in latency…) Data collection in L1 PCs. (which protocol? Routers? Network congestions? Reliability? Fast check at application levels?...) L1 trigger distribution. (Small latency? Latency stability? Broadcasting? …)
Feedback to select computer and network HW Select the transmission protocol
Transmission protocol: UDP vs TCP?
TCP
GOOD:• provides reliable, ordered delivery of astream of bytes. • Flow control.
BAD:• heavy protocol to be implemented in hardware,• each packet needs to be acknowledged and at high rate can become a problem.• The latency is intrinsically not stable.• The packets can be fractioned for transmission optimization reasons
UDP
GOOD:• simple and fast protocol,• easy to be implemented in hardware,• does not require big resources.
BAD:• no reliability, ordering, or data integrity• provides an unreliable service and datagrams may arrive out of order.
Test setupThe software tools for measure the latency provide averages and isn’t clear the contribution to the latency due to the sender and the receiverHW approach: use the oscilloscope to measure the time difference in packet transmission PC
#1PC #2
scope LPTLPT
ETH
[see also R.Fantechi TDAQ 25.5.2011]
Hardware usedPCATE LKRPN0 GPU1
Processor P4 2.4 GHz Xeon 5130 2 GHz (2 processors)
Xeon 5630 2.53 GHz (2 processors, 16 cores)
Cache 512k 4096k 12 MBNIC Intel 82541PI
(bus PCI)Intel 80003E52LAN (bus PCI)
Intel 82574L (PCI express x4)
Driver e1000 e1000 e1000ekernel 2.6.31-
22.generic (ubuntu)
2.6.22-15-generic (kubuntu)
kernel 2.6.18-274.el5 (SL5)
PCATELKRPN0
GPU1
Results: PCATE & GPU1
GPU1
GPU1
PCATE
PCATE
Results: GPU1 & LKRPN0
GPU1
GPU1
LKRPN0
LKRPN0
Results: LKRPN0 & PCATE
PCATE
LKRPN0
LKRPN0
PCATE
Cross check with sockperf
In GPU1->PCATE the result is compatible with the HW measurementsIn the PCATE->GPU1 we have a very big latency
GPU1PCATE
GPU1 PCATE
GPU1 mother board architecture
Processor 1
Processor 2
QUAD PORT GBE
DUAL PORT GBE
IOH IOH
ICH10R
PCI-E x4
QPI
QPI QPI
QPI
ESIx4PCI-
E x1
Test 1
Test 2
“Schizophrenic” test
Same PC, different NIC
GPU1 GPU1
GPU1GPU1
Same PC, Same NIC, different port
What we learned from this first studies?
Both the characteristics of the sender and the receiver are important for the latencyIn particular the sender seems to play an important role using sockperfThe latency increases with the packet dimensionThe frequency of the packet isn’t relevant until the “jump”The “jump” happens at relatively high frequency rate (> 20 kHz)The “jump” probably depends on the NIC/BUS/Chipset more than on CPU/MemoryThe absolute value of the latency depends on the PCs setup and the packet dimension:
Minimum (~300 B packets): ~50 us (recv GPU1 or LKRPN0), ~60 us (recv PCATE)Maximum (~1500B, no jumps if delay is >50us): ~70 us (recv GPU1 or LKRPN0), ~90 us (recv PCATE)
To Do
Use the TALK board to go “faster” (the 10 MHz can’t be reached using PCs)Identify the responsible of the “jumps”:
Same NIC on different PCsDifferent NICs on same PC
Measure the latency with the TELL1 Any suggestions is more than welcome!!!
GPU1 PCATE
LKRPN0