“PCPC Latency measurements”

G.Lamanna, R.Fantechi & J.Kroon(CERN)

TDAQ WG – 8.9.2011

Introduction Why the latency measurement is important

for us?: Primitives transmission to the L0TP through ethernet (Low level protocol, possibility to have switches, small spread in latency…) Data collection in L1 PCs. (which protocol? Routers? Network congestions? Reliability? Fast check at application levels?...) L1 trigger distribution. (Small latency? Latency stability? Broadcasting? …)

Feedback to select computer and network HW Select the transmission protocol

Transmission protocol: UDP vs TCP?

TCP

GOOD:• provides reliable, ordered delivery of astream of bytes. • Flow control.

BAD:• heavy protocol to be implemented in hardware,• each packet needs to be acknowledged and at high rate can become a problem.• The latency is intrinsically not stable.• The packets can be fractioned for transmission optimization reasons

UDP

GOOD:• simple and fast protocol,• easy to be implemented in hardware,• does not require big resources.

BAD:• no reliability, ordering, or data integrity• provides an unreliable service and datagrams may arrive out of order.

Test setupThe software tools for measure the latency provide averages and isn’t clear the contribution to the latency due to the sender and the receiverHW approach: use the oscilloscope to measure the time difference in packet transmission PC

#1PC #2

scope LPTLPT

ETH

[see also R.Fantechi TDAQ 25.5.2011]

Hardware usedPCATE LKRPN0 GPU1

Processor P4 2.4 GHz Xeon 5130 2 GHz (2 processors)

Xeon 5630 2.53 GHz (2 processors, 16 cores)

Cache 512k 4096k 12 MBNIC Intel 82541PI

(bus PCI)Intel 80003E52LAN (bus PCI)

Intel 82574L (PCI express x4)

Driver e1000 e1000 e1000ekernel 2.6.31-

22.generic (ubuntu)

2.6.22-15-generic (kubuntu)

kernel 2.6.18-274.el5 (SL5)

PCATELKRPN0

GPU1

Results: PCATE & GPU1

GPU1

GPU1

PCATE

PCATE

Results: GPU1 & LKRPN0

GPU1

GPU1

LKRPN0

LKRPN0

Results: LKRPN0 & PCATE

PCATE

LKRPN0

LKRPN0

PCATE

Cross check with sockperf

In GPU1->PCATE the result is compatible with the HW measurementsIn the PCATE->GPU1 we have a very big latency

GPU1PCATE

GPU1 PCATE

GPU1 mother board architecture

Processor 1

Processor 2

QUAD PORT GBE

DUAL PORT GBE

IOH IOH

ICH10R

PCI-E x4

QPI

QPI QPI

QPI

ESIx4PCI-

E x1

Test 1

Test 2

“Schizophrenic” test

Same PC, different NIC

GPU1 GPU1

GPU1GPU1

Same PC, Same NIC, different port

What we learned from this first studies?

Both the characteristics of the sender and the receiver are important for the latencyIn particular the sender seems to play an important role using sockperfThe latency increases with the packet dimensionThe frequency of the packet isn’t relevant until the “jump”The “jump” happens at relatively high frequency rate (> 20 kHz)The “jump” probably depends on the NIC/BUS/Chipset more than on CPU/MemoryThe absolute value of the latency depends on the PCs setup and the packet dimension:

Minimum (~300 B packets): ~50 us (recv GPU1 or LKRPN0), ~60 us (recv PCATE)Maximum (~1500B, no jumps if delay is >50us): ~70 us (recv GPU1 or LKRPN0), ~90 us (recv PCATE)

To Do

Use the TALK board to go “faster” (the 10 MHz can’t be reached using PCs)Identify the responsible of the “jumps”:

Same NIC on different PCsDifferent NICs on same PC

Measure the latency with the TELL1 Any suggestions is more than welcome!!!

GPU1 PCATE

LKRPN0