Analyzing InfiniBand Packets - OpenFabrics · OpenFabrics Software User Group Workshop Analyzing...

OpenFabrics

Software

User Group

Workshop

Analyzing InfiniBand Packets

Qian Liu

[email protected]

Advisor: Professor Robert D. Russell

University of New Hampshire

Presentation Overview

• 1. Why analyze IB packets

• 2. How to capture IB packets

• 3. Comparison of IB capture tools

• 4. Our use of the tools to analyze packets

March 19, 2015 #OFAUserGroup 2

1. Why analyze IB packets

• Protocol study, debug, verification, and research

• Monitor IB network performance

• Analyze inter-packet delay (IPD)

• Observe Flow Control and Congestion Control


2. How to capture IB packets

• ibdump…………………………………Software package running on nodes

http://www.mellanox.com/

• CatC analyzer………………………Hardware box inline between ports

http://www.teledynelecroy.com/



• ibdump features

– Software package freely available from Mellanox Technologies

http://downloads.linux.hp.com/downloads/MLNX_OFED/suse/SLES11-

SP2/x86_64/2.2_1.0.1/ibdump-2.0.0-8.x86_64.rpm

– Requires NO physical change to the network

– Runs on an IB host & Captures packets on an IB interface on that host

– Works for all IB data rates: SDR, DDR, QDR, FDR10, FDR

– Dumps a.pcap file which can be loaded by Wireshark http://www.wireshark.com/


Wireshark view of ibdump capture


ibdump

• ibdump limitations

– Cannot capture Flow Control Packets (FCP)

– Packets may get lost if the data rate is high, e.g. FDR (56Gbits/s)

– Works only on Mellanox HCAs

– Doesn’t work between switches because it is software running on nodes

– Max capture size depends on the available host RAM or Disk space

– Inaccurate packet timestamps (in microsecond) (show this next)


Inaccurate microsecond timestamps in ibdump



• CatC analyzer features

– Hardware analyzer from LeCroy

https://www.teledynelecroy.com

– Must be physically placed into an IB link between two IB ports

– Dumps an .ibt file which can be loaded by its IBTracer software

– Works only for SDR (8Gbits/s) data rate

– Works for any type of IB HCAs and switches

– Accurate packet timestamps (in nanosecond)

– Captures ALL packets on the link, including Flow Control Packets (FCP)


https://www.teledynelecroy.com/

CatC analyzer

• Captures packets passing through it in both directions


CatC analyzer Capture


CatC analyzer

• CatC analyzer limitations

– Only works for SDR (8Gbits/s) data rate

– 2GB recording capacity

– Doesn’t dump in .pcap format, so its capture file cannot use Wireshark


3. Comparison between ibdump & CatC analyzer captures First Experiment

One data source is sending 128Mi bytes (MTU = 2k, 65536 packets),

by using RDMA_WRITE, to the receiver via a MLNX SX6036 switch.

Because there is no competing flow, therefore, there should be no

congestion on the link.

ibdump on both sides are running at the same time


3. Comparison between ibdump & CatC analyzer captures First Experiment

Transferring data packets on a SDR (8Gbits/s) link with no congestion,

if each data packet has 2048 bytes payload (MTU is 2k),

The inter-packet time should be around:

2048 bytes * 8 / (8Gbits/s) = 2 us


3. Comparison between ibdump & CatC captures on the receive side

First Experiment


3. ibdump_receiver raw data First Experiment


Comparison of CatC analyzer captures on both sides First Experiment


Comparison of ibdump captures on both sides First Experiment


ibdump_sender and ibdump_receiver raw data First Experiment


3. Comparison between ibdump & CatC analyzer captures Second Experiment

Two data sources, each is sending 128Mi bytes, by using RDMA_WRITE, to

the single receiver via a MLNX SX6036 switch.

The expected inter-packet interval from the same source should be 4 us


Comparison of two sender flows on CatC receive side Second Experiment


Comparison of ibdump sender 1 flow on both sides Second Experiment


ibdump sender 1 flow raw data on both sides Second Experiment


4. Our use of the tools to analyze packets

• 4.1 Flow Control mechanism

• 4.2 Study of the switch buffer size

• 4.3 Study of the tick value


4.1 Flow Control Mechanism

• InfiniBand – Link Layer Flow Control (FC) mechanism

• IB sender will NOT send data packets unless it knows for

sure that the other side of the physical link has enough

buffer to hold the data

• Flow Control Packets (FCPs) are used to report the

available buffer space

• Only CatC analyzer can capture FCPs



• FCP format

• If A sends a FCP to B, then

– FCTBS: total blocks A has sent to B since link initialization

– FCCL: the sum of the total blocks A has received from B, plus the available buffer space in A’s receive buffer

– Both numbers are increasing monotonically, modulo 4096

– One block is 64 bytes of buffer space



• Experiment:

– A sender is sending 128Mi bytes of data to a receiver, using RDMA_WRITE

– MTU = 2k, 65536 data packets

– Each packet is at least 2048 + 8 + 12 + 6 = 2074 bytes.

– Each packet occupies = 33 FC blocks



• Starting FCCL/FCTBS before A (Tx) sends data packets to B (Rx)

A has sent 1404 blocks to B

A receives a FCP from B, in which the FCCL value is 3206

3206 = total blocks B has received from A + the available receive buffer space in B

3206 – 1404 >> 33, based on this calculation, A is able to send a data packet



• FCCL value update -> means one or more blocks are released in B’s

receive buffer



• FCTBS value update



– Before A sends data packets to B, the starting FCTBS value is 1404

……………………………

– The latest FCTBS value is 2262

– (2262 - 1404) / 33 = 26 data packets have been sent from A to B


4.2 Study of switch buffer size

• Object:

MLNX SX6036 FDR switch

Use the CatC analyzer to determine the switch buffer size

Assumption:

1. input-queued switch

2. shared buffer per port, divided by the available Virtual Lanes (VLs)



The buffer size is an indicator of the latency a program may experience

SDR 1 and SDR 2, two senders are sending data to a SDR receiver

MTU 2k, data transmission is on VL0 (Start SDR 2 later than SDR 1)

1. at the very beginning, each SDR sender can inject packets in 2us

2. when congestion occurs, each SDR sender can only inject packets in 4us




• Buffer space on each port is not full

• Packets can be put in 2us interval

SDR 1

SDR 2

SDR R

Switch

VL Buffer

VL Buffer



• Buffer space on each port is full

• Senders have to wait until there are enough buffer space on switch

port to hold the data packets

SDR 1

SDR 2

SDR R

Switch

VL Buffer

VL Buffer


A2: The first data packet of SDR 2 (SDR 2 is started later than SDR 1)

B1: The first SDR 1 data packet whose inter-packet interval on its sending side is 4us



On Mellanox SX6036 switch,

By counting the number of the green packets in the 2nd phase,

the determined switch input VL buffer space is around 32Ki bytes.

With configuration of 4 VLs, 4 * 32Ki = 128Ki bytes for each input port


4.3 Study of tick value

Congestion Indicator (counter) PortXmitWait:

Port counter that is used to indicate the "number of ticks during

which selected port had data to transmit but none was sent during

the entire tick either because of insufficient credits or due to

lack of arbitration"



PortXmitWait:

What is the tick?

Tick indicates the node’s sampling clock interval:

encoding value * symbol time

symbol time:

the time required to transmit an 8 bit data quantity onto a physical lane

(SDR symbol time 4ns)

encoding value:

multiple of the symbol time. 1 ~ 256

# perfquery –c LID Port_Number



Both buffers are empty

Buffers are full

Buffers are filling up


SDR 1

SDR 2

SDR R

Switch

PortXmitWait unchanged

SDR 1

SDR 2

SDR R

Switch

PortXmitWait unchanged

SDR 1

SDR 2

SDR R

Switch

PortXmitWait incremented


A2: Time when SDR R starts receiving packets from both competing flows

B1: Time when the inter-packet intervals on each sender side go up to 4us

L: Time when SDR R receives the last SDR 1 data packet



• Tick =

• Duration of the Congestion = TIMEB1-L - TIMEregular



• Congestion time

MLNX MT26428 QDR CA

encoding value = 31 = 0x1F

# perfquery –c LID 1

Tick……………………..0x1F


Acknowledgement

I would like to thank for their support

- My advisor, Professor Robert D. Russell

- National Science Foundation Grant OCI-1127228

- Software Forge, Inc. -- for the loan of the CatC analyzers

- University of New Hampshire InterOperability Lab (UNH IOL)


#OFSUserGroup

OpenFabrics Software

User Group Workshop

Thank You

Date post:	05-Aug-2018
Category:	Documents
Upload:	buixuyen
View:	225 times
Download:	0 times

Analyzing InfiniBand Packets - OpenFabrics · OpenFabrics Software User Group Workshop Analyzing...

Documents