Download - Timing Analysis of a Linux-Based CAN-to-CAN Gateway

Timing Analysis of a Linux-BasedCAN-to-CAN Gateway

Michal Sojka1, Pavel Píša1, Ondrej Špinka1,Oliver Hartkopp2, Zdenek Hanzálek1

1Czech Technical University in PragueFaculty of Electrical Engineering

2Volkswagen Group Research

13th Real-Time Linux WorkshopOctober 22, Prague

Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 1 / 26

Outline

Motivation

Testbed Description

Analysis Results

Conclusions


Motivation

CAN Bus

I Intended to interconnect microcontrollers for shortdistances.

I Network with deterministic medium access ⇒ real-timeapplications.

I Heavily used in automotive industry.I Bit rates up to 1 Mbit/s.I Every frame can carry up to 8 bytes of data.I Frames are identified by frame IDs (11 or 29 bits).


Motivation

Rapid prototyping with CAN

I Goal: Quickly interconnect devices/subsystems frommultiple vendors.

I Frame ID clashes cause problems – devices cannot beconnected together.

I A need for CAN-to-CAN gateways to route (and modify)messages between multiple buses.


Motivation

SocketCAN

I Open-source project, official CAN-bus support for Linuxkernel

I Initiated by Volkswagen research – first open sourceproject where Volkswagen participates

I Core developers also from a few other companiesI Contributors from many other peopleI In mainline Linux since 2.6.25I Based on standard Linux networking infrastructureI http://socketcan.berlios.de


http://socketcan.berlios.de

Motivation

SocketCAN-based Gateway

I Routes CAN frames between multiple CAN buses.I Frames can be modified (ID, length, data)I Modifications: SET, AND, OR, XOR, CRCI Frames can be filtered based on ID/mask pairs.I Implemented as a Linux kernel moduleI Easy to use:

cangw -A -s can0 -d can1


Motivation

Motivation of our Work

I Adding gateways to real-time network can violate real-timerequirements (deadlines).

I How are the routed frames delayed by the gateway?I What influences the delays (latencies)?

I Traffic patterns,I Additional load,I Gateway configuration,I Linux kernel version.

I Where are the limits of the SocketCAN-based gateway?

I Nice way to measure temporal properties of the Linuxkernel.

I Some of our results apply not only for CAN, but also forLinux networking subsystem in general.


Motivation

Motivation of our Work

I Adding gateways to real-time network can violate real-timerequirements (deadlines).

I How are the routed frames delayed by the gateway?I What influences the delays (latencies)?

I Traffic patterns,I Additional load,I Gateway configuration,I Linux kernel version.

I Where are the limits of the SocketCAN-based gateway?

I Nice way to measure temporal properties of the Linuxkernel.

I Some of our results apply not only for CAN, but also forLinux networking subsystem in general.


Testbed Description

Testbed Setup

I Gateway – PowerPC MPC5200 (Busybox)I PC – Pentium 4, Kvaser PCI quad-CAN SJA1000-based


Testbed Description

Testbed Setup

I Gateway – PowerPC MPC5200 (Busybox)I PC – Pentium 4, Kvaser PCI quad-CAN SJA1000-based


Testbed Description

Measured Latencies

CAN bus 0

CAN bus 1

time

msg 1

Duration

msg 1'

CAN gateway(Linux)

GW latency

RX timestamp 1 RX timestamp 2

Total latency

I Same clock for bothtimestamps

I SO_TIMESTAMPNSsocket option

I Clock source = TSC


Testbed Description

Tested Scenarios

I Traffic patterns:I one message at a time,I 50% bus load,I flood (almost 100% bus load)

I Additional loads:I CPU load – hackbenchI Ethernet load – ping -f -s 6000

I Versions:I Kernel: 2.6.33, 2.6.33-rt29, 2.6.36, 3.0.4, 3.0.4-rt14I Latest SocketCAN from SVN (rev 1199 + fixes)

I All combinations of the above.


Analysis Results

The Simplest GW Configuration

I Experiment:I Frames with payload of 2 bytesI Message transmission time: 60µs


Analysis Results

Presentation of Results – Histogram

0

1000

2000

3000

4000

0 20 40 60 80 100

Fram

es

GW latency [µs]

Histogram

0 2000 4000 6000 8000

10000

0 50 100 150 200

Fram

es

Cumulative histogram

0 2000 4000 6000 8000

10000

0 50 100 150 200

Fram

es

Backward cumulative histogram

1

10

100

1000

10000

0 50 100 150 200

Fram

es

Latency profile

I What is theworst-case latency?

I How many frameshad latency greaterthen 60µs?

I Backward-cumulativehistogram.

I Logarithmic scale.I Answer: 10 frames


Analysis Results

Presentation of Results – Cumulative Histogram

0

1000

2000

3000

4000

0 20 40 60 80 100

Fram

es

GW latency [µs]

Histogram

0 2000 4000 6000 8000

10000

0 50 100 150 200

Fram

es


0 2000 4000 6000 8000

10000

0 50 100 150 200

Fram

es


1

10

100

1000

10000

0 50 100 150 200

Fram

es

Latency profile






Analysis Results

Presentation of Results – Backward-Cumulative H.

0

1000

2000

3000

4000

0 20 40 60 80 100

Fram

es

GW latency [µs]

Histogram

0 2000 4000 6000 8000

10000

0 50 100 150 200

Fram

es


0 2000 4000 6000 8000

10000

0 50 100 150 200

Fram

es


1

10

100

1000

10000

0 50 100 150 200

Fram

es

Latency profile






Analysis Results

Presentation of Results – Latency Profile

0

1000

2000

3000

4000

0 20 40 60 80 100

Fram

es

GW latency [µs]

Histogram

0 2000 4000 6000 8000

10000

0 50 100 150 200

Fram

es


0 2000 4000 6000 8000

10000

0 50 100 150 200

Fram

es


1

10

100

1000

10000

0 50 100 150 200

Fram

es

Latency profile






Analysis Results

Presentation of Results – Multiple plots

0

1000

2000

3000

4000

0 20 40 60 80 100

Fram

es

GW latency [µs]

No loadCPU load

0 2000 4000 6000 8000

10000

0 50 100 150 200

Fram

es

No loadCPU load

0 2000 4000 6000 8000

10000

0 50 100 150 200

Fram

es

No loadCPU load

1

10

100

1000

10000

0 50 100 150 200

Fram

es

No loadCPU load






Analysis Results

Latency Profile vs. Time Plot

1

10

100

1000

10000

0 20 40 60 80 100 120 140

Late

ncy

pro

file

[fra

mes]

GW latency [µs]

0

20

40

60

80

100

120

140

2.5 3 3.5 4 4.5 5 5.5G

W late

ncy

[µ

s]Experiment time [s]


Analysis Results

Measurement PrecisionOur test-bed (PC) versus CANalyzer.

1

10

100

1000

TX-duration=84

0 50 100 150 200 250

Late

ncy

pro

file

[fra

mes]

Total latency [µs]

PCCANalyzer

I Difference 10µs is sufficient for us.I Systematic error?I CANanalyzer has only resolution of 10µs.


Analysis Results

Measurement PrecisionInfluence of Ethernet load generator on measurement precision. No GW involved!


Analysis Results

Measurement PrecisionInfluence of Ethernet load generator on measurement precision. No GW involved!

1

10

100

1000

10000

31 0 50 100 150 200 250 300

Late

ncy

pro

file

[fra

mes]

TX+RX overhead [µs]i.e. tkernel RX - tbefore send - tTX duration

No loadEthernet load

I TX+RX overhead (interfaces can0 and can1 on the PC)I Desired result: vertical lines at the same positionI The error is much smaller than latencies observed for the

gateway.


Analysis Results

Batched Processing of Frames

1

10

100

1000

10000

0 20 40 60 80 100 120 140

Late

ncy

pro

file

[fra

mes]

GW latency [µs]

One frame at a timeFlood

I Soft-IRQ processes packets in batches (if possible).I Decreases average latency.


Analysis Results

Effect of Loading the Gateway – CPU

1

10

100

1000

10000

0 50 100 150 200 250 300

Late

ncy

pro

file

[fra

mes]

GW latency [µs]

No loadCPU load

I Hackbench


Analysis Results

Effect of Loading the Gateway – Ethernet

1

10

100

1000

10000

0 250 500 750 1000 1250 1500 1750 2000 2250 2500 2750 3000

Late

ncy

pro

file

[fra

mes]

GW latency [µs]

No loadCPU load

Ethernet load

I ping -f -s 60000


Analysis Results

Frame Filtering

1

10

100

1000

10000

0 200 400 600 800 1000

Late

ncy

pro

file

[fra

mes]

Different number of filters, only the last one matches

List length 1List length 128List length 256List length 384List length 512List length 768

List length 1024List length 1536List length 2048

1

10

100

1000

10000

0 200 400 600 800 1000

Late

ncy

pro

file

[fra

mes]

GW latency [µs]

2048 SFF filters, different frame IDs

Frame ID 1Frame ID 256Frame ID 512Frame ID 768

Frame ID 1024

Frame ID 1536Frame ID 2048

I Two filter implementations:list traversal, array indexing

I More than 80 list entries ⇒the delay increases faster(cache trashing)


Analysis Results

Frame Modifications

1

10

100

1000

10000

0 50 100 150 200

Late

ncy

pro

file

[fra

mes]

GW latency [µs]

No modifications2 modifications4 modifications

4 modifications + XOR checksum4 modifications + CRC8 checksum


Analysis Results

Differences between Kernels – non-rt

1

10

100

1000

10000

0 20 40 60 80 100 120 140

Late

ncy

pro

file

[fra

mes]

GW latency [µs]

2.6.33.72.6.36.23.0.4

I The overhead increases over timeI make oldconfig


Analysis Results

Differences between Kernels – rt-preempt

1

10

100

1000

10000

0 50 100 150 200 250 300 350

Late

ncy

pro

file

[fra

mes]

GW latency [µs]

2.6.33.72.6.36.23.0.42.6.33.7-rt29


Analysis Results

Differences between Kernels – rt-preempt

1

10

100

1000

10000

0 50 100 150 200 250 300 350

Late

ncy

pro

file

[fra

mes]

GW latency [µs]

2.6.33.72.6.36.23.0.42.6.33.7-rt293.0.4-rt143.0.7-rt20


Analysis Results

Problem (bug?) in rt-preempt kernelsGW kernel 3.0.4-rt14, Traffic oneatatime, Load none

1

10

100

1000

10000

0.5 1 2 5 10 20 50

Late

ncy

pro

file

[fra

mes]

GW latency [ms]

Frame id 0Frame id 255Frame id 511Frame id 767

I Huge latencies in rt-preempt kernel. Conditions: 2048 EFFfilters, GW kernel: 3.0.4-rt14, traffic: one frame at a time,load: none, payload: 2 bytes.

I Appears also in 2.6.33.7-rt29I High latency repeats periodically every 1 second


Analysis Results

Problem (bug?) in rt-preempt kernelsGW kernel 3.0.4-rt14, Traffic oneatatime, Load none

0.1

1

10

100

0 1 2 3 4 5 6 7 8

GW

late

ncy

[m

s]

Experiment time [s]

Frame ID 511

I Huge latencies in rt-preempt kernel. Conditions: 2048 EFFfilters, GW kernel: 3.0.4-rt14, traffic: one frame at a time,load: none, payload: 2 bytes.

I Appears also in 2.6.33.7-rt29I High latency repeats periodically every 1 second


Analysis Results

User-Space Gateway

Traffic: one at a time

1

10

100

1000

10000

0 100 200 300 400 500

Late

ncy

pro

file

[fra

mes]

GW Latency [µs]

Kernel GWUserspace GW

Traffic: flood

1

10

100

1000

10000

0.01 0.1 1 10 100 1000 10000 100000

Late

ncy

pro

file

[fra

mes]

GW latency [ms]

Kernel GWUser GWKernel GW, 33-rtUser GW, 33-rt


Analysis Results

Results processing

I More than 600 experimentsI HTML-based navigation on results


Analysis Results

Results processing

I More than 600 experimentsI HTML-based navigation on results


Conclusions

Conclusion

I SocketCAN-based gateway is a reliable solution for CANbus routing.

I Kernel-space solution performs better than a user-spaceone.

I There are things to avoid:I Combining CAN traffic with Ethernet traffic.I Use of too many filters.

I The gateway will be merged in 3.2 kernel.

I Nice way of measuring Linux kernel temporal propertiesfrom outside.

I https://rtime.felk.cvut.cz/can/benchmark/2.1/

I https://rtime.felk.cvut.cz/gitweb/can-benchmark.git

Questions?


https://rtime.felk.cvut.cz/can/benchmark/2.1/

https://rtime.felk.cvut.cz/gitweb/can-benchmark.git

Conclusions

Conclusion








Questions?




Conclusions

Conclusion








Questions?Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 26 / 26

