© 2011 Toshiba Corporation

Evaluation of TIPCMitsuhiro KimuraTOSHIBA CORPORATION

CELF Japan Technical Jamboree #37, May 20, 2011

2© 2010 Toshiba Corporation© 2011 Toshiba Corporation

Background

• Integrating one more SoC in the embedded system for adding more functions.

• We will discuss the interconnection of two SoCs:– It needs to choose a bus from available interface buses in

SoCs.– Ethernet is more commonly available.

• Required a quick and lightweight protocol stack on Ethernet.

SoC A

SoC BProduct

SoC: System-on-a-chip

3© 2010 Toshiba Corporation© 2011 Toshiba Corporation

TIPC : Transparent Inter-Process Communication

• The protocol specifically designed for cluster communication – quick and lightweight

• Supposing little packet loss and infrequent retransmission– Socket I/F

• sockets corresponding TCP, UDP.– Built in Linux Kernel.

TIPC:Kernel: 2.6.35 ~

2.0.01.6, 1.72.6.16 ~ 34

1.5.122.6.9 ~ 14

1.4.32.4, 2.6

5/14 1.7.7

4© 2010 Toshiba Corporation© 2011 Toshiba Corporation

Evaluation environment

• Interconnection of SoCs in embedded systems is similar to cluster connections in that both of them are little packet loss intra-communications.

• Testing SoC’s used for embedded systems– (reference [2], evaluation with Xeon, gigabit ethernet)

• Benchmarking throughput as well as CPU load– MIPS32®24K® 533MHz ×2, 100Mbps Ethernet– Linux Kernel 2.6.20.19, TIPC Version 1.7.7-rc1

SoC A

BoardSoC B

5© 2010 Toshiba Corporation© 2011 Toshiba Corporation

Test cases

• (1) Max throughput– Send data in one direction.– Measure throughput and CPU load

per message size• (2) Latency

– Data flows in both directions alternately in a ping-pong fashion.

– Measure the roundtrip time per message size.

• (3) CPU load for Streaming– Send data in one direction at a constant rate.– Measure the CPU load.

6© 2010 Toshiba Corporation© 2011 Toshiba Corporation

(1) Max throughput, CPU load

• Message size: 512, 1024, 1536, … 40448Byte– Watch a change depending on message size.– Compare throughput and CPU load of TIPC with TCP/IP.

• Measure the time of send()/recv() 10,000 times.

512×n Byte

SoC A SoC B

7© 2010 Toshiba Corporation© 2011 Toshiba Corporation

(1) Max throughput

• TCP/IP < TIPC– Header: TCP/IP:40Byte, TIPC:24Byte, MTU:1500Byte– Small header high transfer efficiency

77

79

81

83

85

87

89

512 10624 20736 30848 40960

Thro

ughp

ut [M

bps]

Message size [Byte]

Message size - Throughput

8© 2010 Toshiba Corporation© 2011 Toshiba Corporation

(1) Max throughput

• TIPC throughput drops in case one more packet needs to be sent.

8383.5

8484.5

8585.5

8686.5

8787.5

88

2048 2432 2816 3200 3584

Thro

ughp

ut [M

bps]

Message size[Byte]

Message size - Throughput 2952 Byte= (1500-24)×2

3449

packet: 23

9© 2010 Toshiba Corporation© 2011 Toshiba Corporation

Reason of TIPC throughput drop

• 1500Byte – 24Byte = 1476Byte.2953Byte = 1476Byte × 2 + 1Byte

TIPC:

TCP/IP:

1460 14601460SoC A SoC B1460 1460

send() send()

• 1500Byte – 40Byte = 1460Byte.

1476 14761SoC A SoC B1476 14761

send() send()

10© 2010 Toshiba Corporation© 2011 Toshiba Corporation

(1) CPU load

• CPU load of TIPC is less than TCP/IP

0

10

20

30

40

50

512 10624 20736 30848 40960

CPU

load

(%)

Message size[Byte]

send

0

20

40

60

80

100

512 10624 20736 30848 40960

CPU

load

(%)

Message size[Byte]

recv

11© 2010 Toshiba Corporation© 2011 Toshiba Corporation

Packets observance

• TIPC performance in case of small size packets– Counting RX/TX packets.

SoC A SoC BMessage

eth0 Link encap:Ethernet HWaddr XX:XX:XX:XX:XX:XXinet addr:X.X.X.X Bcast:X.X.X.X Mask:X.X.X.XUP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1RX packets:0 errors:0 dropped:0 overruns:0 frame:0TX packets:0 errors:0 dropped:0 overruns:0 carrier:0collisions:0 txqueuelen:1000RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

ifconfig

12© 2010 Toshiba Corporation© 2011 Toshiba Corporation

TCP/IP TIPC

Message Packets: TCP/IP < TIPC

0

50000

100000

150000

200000

250000

300000

0 50 100 150 200

Num

ber o

f pac

kets

Time [sec]

0

50000

100000

150000

200000

250000

300000

0 20 40 60 80

Num

ber o

f pac

kets

Time [sec]

• Message size: 512 Byte• TCP/IP small packets coalesce.

Send packetsRecv packets

13© 2010 Toshiba Corporation© 2011 Toshiba Corporation

TCP/IP TIPC

ACK: TCP/IP > TIPC• Message size: 4096 Byte• TIPC is less frequent ACK.

0

10000

20000

30000

40000

50000

0 20 40 60 80

Num

ber o

f pac

kets

Time [sec]

ACK

05000

1000015000200002500030000350004000045000

0 20 40 60 80

Num

ber o

f pac

kets

Time [sec]

ACK

Low CPU load

Send packetsRecv packets

14© 2010 Toshiba Corporation© 2011 Toshiba Corporation

(2) Latency

• TCP/IP’s latency nearly equals TIPC’s.

00.0010.0020.0030.0040.0050.0060.0070.0080.009

512 10624 20736 30848 40960

Late

ncy[

sec]

Message size [Byte]

Message size - Latency SoC B

SoC A

・・・

Latency

15© 2010 Toshiba Corporation© 2011 Toshiba Corporation

CPU load[%]

(3) CPU load in sending streaming data.

• Data transmission rate: 25Mbps

TCP/IP TIPC TCP/IP - TIPCSoC A(send) 9.16 5.18 3.98SoC B(recv) 19.25 10.25 9.00

SoC BSoC A

nanosleep()80msec

256KByte

16© 2010 Toshiba Corporation© 2011 Toshiba Corporation

Use TIPC (1/2)

• TIPC doesn’t support SO_SNDTIMEO. Blocking-send is hard to use. Nonblocking-send repeat. CPU load increases. (test program: ~5%)

17© 2010 Toshiba Corporation© 2011 Toshiba Corporation

Use TIPC (2/2)

• Processes/threads issue a lot of send request TIPC send queue is filling up.

(congestion control)– Increase processes waiting for queue to decrease.– Even if queue decreases, and processes get up, wait

again without issue a send request.– Need to schedule between processes.

TIPCsend queue

Send packets

20© 2010 Toshiba Corporation© 2011 Toshiba Corporation

Conclusion

• TIPC is faster than TCP/IP– Smaller header size– TCP/IP is faster in case of small message size.

• Merging small messages would improve the throughput.

• TIPC has lower CPU load than TCP/IP– Less frequent ACK (other reasons would exist)

• Verified that TIPC is useful for embedded systems.– (e.g. Reducing CPU load when transferring streaming

data.)

21© 2010 Toshiba Corporation© 2011 Toshiba Corporation

Reference• Reference:1. TIPC Home Page, http://tipc.sourceforge.net/2. Florian Westphal, 2007, TIPC Analysis and Optimization,

http://webuser.hs-furtwangen.de/~reich/AdvancedMiddlewareWorkshop.SS07

3. Mitsuhiro Kimura: “Application of TIPC to communicate between CPUs in an embedded device and performance comparison to TCP/IP”, The 73rd National Convention of IPSJ 6G-3 1-43 – 1-44 (2011-3)