Date post: | 26-May-2015 |
Category: |
Documents |
Upload: | michal-sojka |
View: | 1,842 times |
Download: | 2 times |
Timing Analysis of a Linux-BasedCAN-to-CAN Gateway
Michal Sojka1, Pavel Píša1, Ondrej Špinka1,Oliver Hartkopp2, Zdenek Hanzálek1
1Czech Technical University in PragueFaculty of Electrical Engineering
2Volkswagen Group Research
13th Real-Time Linux WorkshopOctober 22, Prague
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 1 / 26
Outline
Motivation
Testbed Description
Analysis Results
Conclusions
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 2 / 26
Motivation
CAN Bus
I Intended to interconnect microcontrollers for shortdistances.
I Network with deterministic medium access ⇒ real-timeapplications.
I Heavily used in automotive industry.I Bit rates up to 1 Mbit/s.I Every frame can carry up to 8 bytes of data.I Frames are identified by frame IDs (11 or 29 bits).
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 3 / 26
Motivation
Rapid prototyping with CAN
I Goal: Quickly interconnect devices/subsystems frommultiple vendors.
I Frame ID clashes cause problems – devices cannot beconnected together.
I A need for CAN-to-CAN gateways to route (and modify)messages between multiple buses.
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 4 / 26
Motivation
SocketCAN
I Open-source project, official CAN-bus support for Linuxkernel
I Initiated by Volkswagen research – first open sourceproject where Volkswagen participates
I Core developers also from a few other companiesI Contributors from many other peopleI In mainline Linux since 2.6.25I Based on standard Linux networking infrastructureI http://socketcan.berlios.de
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 5 / 26
Motivation
SocketCAN-based Gateway
I Routes CAN frames between multiple CAN buses.I Frames can be modified (ID, length, data)I Modifications: SET, AND, OR, XOR, CRCI Frames can be filtered based on ID/mask pairs.I Implemented as a Linux kernel moduleI Easy to use:
cangw -A -s can0 -d can1
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 6 / 26
Motivation
Motivation of our Work
I Adding gateways to real-time network can violate real-timerequirements (deadlines).
I How are the routed frames delayed by the gateway?I What influences the delays (latencies)?
I Traffic patterns,I Additional load,I Gateway configuration,I Linux kernel version.
I Where are the limits of the SocketCAN-based gateway?
I Nice way to measure temporal properties of the Linuxkernel.
I Some of our results apply not only for CAN, but also forLinux networking subsystem in general.
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 7 / 26
Motivation
Motivation of our Work
I Adding gateways to real-time network can violate real-timerequirements (deadlines).
I How are the routed frames delayed by the gateway?I What influences the delays (latencies)?
I Traffic patterns,I Additional load,I Gateway configuration,I Linux kernel version.
I Where are the limits of the SocketCAN-based gateway?
I Nice way to measure temporal properties of the Linuxkernel.
I Some of our results apply not only for CAN, but also forLinux networking subsystem in general.
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 7 / 26
Testbed Description
Testbed Setup
I Gateway – PowerPC MPC5200 (Busybox)I PC – Pentium 4, Kvaser PCI quad-CAN SJA1000-based
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 8 / 26
Testbed Description
Testbed Setup
I Gateway – PowerPC MPC5200 (Busybox)I PC – Pentium 4, Kvaser PCI quad-CAN SJA1000-based
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 8 / 26
Testbed Description
Measured Latencies
CAN bus 0
CAN bus 1
time
msg 1
Duration
msg 1'
CAN gateway(Linux)
GW latency
RX timestamp 1 RX timestamp 2
Total latency
I Same clock for bothtimestamps
I SO_TIMESTAMPNSsocket option
I Clock source = TSC
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 9 / 26
Testbed Description
Tested Scenarios
I Traffic patterns:I one message at a time,I 50% bus load,I flood (almost 100% bus load)
I Additional loads:I CPU load – hackbenchI Ethernet load – ping -f -s 6000
I Versions:I Kernel: 2.6.33, 2.6.33-rt29, 2.6.36, 3.0.4, 3.0.4-rt14I Latest SocketCAN from SVN (rev 1199 + fixes)
I All combinations of the above.
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 10 / 26
Analysis Results
The Simplest GW Configuration
I Experiment:I Frames with payload of 2 bytesI Message transmission time: 60µs
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 11 / 26
Analysis Results
Presentation of Results – Histogram
0
1000
2000
3000
4000
0 20 40 60 80 100
Fram
es
GW latency [µs]
Histogram
0 2000 4000 6000 8000
10000
0 50 100 150 200
Fram
es
Cumulative histogram
0 2000 4000 6000 8000
10000
0 50 100 150 200
Fram
es
Backward cumulative histogram
1
10
100
1000
10000
0 50 100 150 200
Fram
es
Latency profile
I What is theworst-case latency?
I How many frameshad latency greaterthen 60µs?
I Backward-cumulativehistogram.
I Logarithmic scale.I Answer: 10 frames
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 12 / 26
Analysis Results
Presentation of Results – Cumulative Histogram
0
1000
2000
3000
4000
0 20 40 60 80 100
Fram
es
GW latency [µs]
Histogram
0 2000 4000 6000 8000
10000
0 50 100 150 200
Fram
es
Cumulative histogram
0 2000 4000 6000 8000
10000
0 50 100 150 200
Fram
es
Backward cumulative histogram
1
10
100
1000
10000
0 50 100 150 200
Fram
es
Latency profile
I What is theworst-case latency?
I How many frameshad latency greaterthen 60µs?
I Backward-cumulativehistogram.
I Logarithmic scale.I Answer: 10 frames
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 12 / 26
Analysis Results
Presentation of Results – Backward-Cumulative H.
0
1000
2000
3000
4000
0 20 40 60 80 100
Fram
es
GW latency [µs]
Histogram
0 2000 4000 6000 8000
10000
0 50 100 150 200
Fram
es
Cumulative histogram
0 2000 4000 6000 8000
10000
0 50 100 150 200
Fram
es
Backward cumulative histogram
1
10
100
1000
10000
0 50 100 150 200
Fram
es
Latency profile
I What is theworst-case latency?
I How many frameshad latency greaterthen 60µs?
I Backward-cumulativehistogram.
I Logarithmic scale.I Answer: 10 frames
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 12 / 26
Analysis Results
Presentation of Results – Latency Profile
0
1000
2000
3000
4000
0 20 40 60 80 100
Fram
es
GW latency [µs]
Histogram
0 2000 4000 6000 8000
10000
0 50 100 150 200
Fram
es
Cumulative histogram
0 2000 4000 6000 8000
10000
0 50 100 150 200
Fram
es
Backward cumulative histogram
1
10
100
1000
10000
0 50 100 150 200
Fram
es
Latency profile
I What is theworst-case latency?
I How many frameshad latency greaterthen 60µs?
I Backward-cumulativehistogram.
I Logarithmic scale.I Answer: 10 frames
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 12 / 26
Analysis Results
Presentation of Results – Multiple plots
0
1000
2000
3000
4000
0 20 40 60 80 100
Fram
es
GW latency [µs]
No loadCPU load
0 2000 4000 6000 8000
10000
0 50 100 150 200
Fram
es
No loadCPU load
0 2000 4000 6000 8000
10000
0 50 100 150 200
Fram
es
No loadCPU load
1
10
100
1000
10000
0 50 100 150 200
Fram
es
No loadCPU load
I What is theworst-case latency?
I How many frameshad latency greaterthen 60µs?
I Backward-cumulativehistogram.
I Logarithmic scale.I Answer: 10 frames
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 12 / 26
Analysis Results
Latency Profile vs. Time Plot
1
10
100
1000
10000
0 20 40 60 80 100 120 140
Late
ncy
pro
file
[fra
mes]
GW latency [µs]
0
20
40
60
80
100
120
140
2.5 3 3.5 4 4.5 5 5.5G
W late
ncy
[µ
s]Experiment time [s]
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 13 / 26
Analysis Results
Measurement PrecisionOur test-bed (PC) versus CANalyzer.
1
10
100
1000
TX-duration=84
0 50 100 150 200 250
Late
ncy
pro
file
[fra
mes]
Total latency [µs]
PCCANalyzer
I Difference 10µs is sufficient for us.I Systematic error?I CANanalyzer has only resolution of 10µs.
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 14 / 26
Analysis Results
Measurement PrecisionInfluence of Ethernet load generator on measurement precision. No GW involved!
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 15 / 26
Analysis Results
Measurement PrecisionInfluence of Ethernet load generator on measurement precision. No GW involved!
1
10
100
1000
10000
31 0 50 100 150 200 250 300
Late
ncy
pro
file
[fra
mes]
TX+RX overhead [µs]i.e. tkernel RX - tbefore send - tTX duration
No loadEthernet load
I TX+RX overhead (interfaces can0 and can1 on the PC)I Desired result: vertical lines at the same positionI The error is much smaller than latencies observed for the
gateway.
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 15 / 26
Analysis Results
Batched Processing of Frames
1
10
100
1000
10000
0 20 40 60 80 100 120 140
Late
ncy
pro
file
[fra
mes]
GW latency [µs]
One frame at a timeFlood
I Soft-IRQ processes packets in batches (if possible).I Decreases average latency.
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 16 / 26
Analysis Results
Effect of Loading the Gateway – CPU
1
10
100
1000
10000
0 50 100 150 200 250 300
Late
ncy
pro
file
[fra
mes]
GW latency [µs]
No loadCPU load
I Hackbench
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 17 / 26
Analysis Results
Effect of Loading the Gateway – Ethernet
1
10
100
1000
10000
0 250 500 750 1000 1250 1500 1750 2000 2250 2500 2750 3000
Late
ncy
pro
file
[fra
mes]
GW latency [µs]
No loadCPU load
Ethernet load
I ping -f -s 60000
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 18 / 26
Analysis Results
Frame Filtering
1
10
100
1000
10000
0 200 400 600 800 1000
Late
ncy
pro
file
[fra
mes]
Different number of filters, only the last one matches
List length 1List length 128List length 256List length 384List length 512List length 768
List length 1024List length 1536List length 2048
1
10
100
1000
10000
0 200 400 600 800 1000
Late
ncy
pro
file
[fra
mes]
GW latency [µs]
2048 SFF filters, different frame IDs
Frame ID 1Frame ID 256Frame ID 512Frame ID 768
Frame ID 1024
Frame ID 1536Frame ID 2048
I Two filter implementations:list traversal, array indexing
I More than 80 list entries ⇒the delay increases faster(cache trashing)
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 19 / 26
Analysis Results
Frame Modifications
1
10
100
1000
10000
0 50 100 150 200
Late
ncy
pro
file
[fra
mes]
GW latency [µs]
No modifications2 modifications4 modifications
4 modifications + XOR checksum4 modifications + CRC8 checksum
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 20 / 26
Analysis Results
Differences between Kernels – non-rt
1
10
100
1000
10000
0 20 40 60 80 100 120 140
Late
ncy
pro
file
[fra
mes]
GW latency [µs]
2.6.33.72.6.36.23.0.4
I The overhead increases over timeI make oldconfig
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 21 / 26
Analysis Results
Differences between Kernels – rt-preempt
1
10
100
1000
10000
0 50 100 150 200 250 300 350
Late
ncy
pro
file
[fra
mes]
GW latency [µs]
2.6.33.72.6.36.23.0.42.6.33.7-rt29
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 22 / 26
Analysis Results
Differences between Kernels – rt-preempt
1
10
100
1000
10000
0 50 100 150 200 250 300 350
Late
ncy
pro
file
[fra
mes]
GW latency [µs]
2.6.33.72.6.36.23.0.42.6.33.7-rt293.0.4-rt143.0.7-rt20
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 22 / 26
Analysis Results
Problem (bug?) in rt-preempt kernelsGW kernel 3.0.4-rt14, Traffic oneatatime, Load none
1
10
100
1000
10000
0.5 1 2 5 10 20 50
Late
ncy
pro
file
[fra
mes]
GW latency [ms]
Frame id 0Frame id 255Frame id 511Frame id 767
I Huge latencies in rt-preempt kernel. Conditions: 2048 EFFfilters, GW kernel: 3.0.4-rt14, traffic: one frame at a time,load: none, payload: 2 bytes.
I Appears also in 2.6.33.7-rt29I High latency repeats periodically every 1 second
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 23 / 26
Analysis Results
Problem (bug?) in rt-preempt kernelsGW kernel 3.0.4-rt14, Traffic oneatatime, Load none
0.1
1
10
100
0 1 2 3 4 5 6 7 8
GW
late
ncy
[m
s]
Experiment time [s]
Frame ID 511
I Huge latencies in rt-preempt kernel. Conditions: 2048 EFFfilters, GW kernel: 3.0.4-rt14, traffic: one frame at a time,load: none, payload: 2 bytes.
I Appears also in 2.6.33.7-rt29I High latency repeats periodically every 1 second
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 23 / 26
Analysis Results
User-Space Gateway
Traffic: one at a time
1
10
100
1000
10000
0 100 200 300 400 500
Late
ncy
pro
file
[fra
mes]
GW Latency [µs]
Kernel GWUserspace GW
Traffic: flood
1
10
100
1000
10000
0.01 0.1 1 10 100 1000 10000 100000
Late
ncy
pro
file
[fra
mes]
GW latency [ms]
Kernel GWUser GWKernel GW, 33-rtUser GW, 33-rt
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 24 / 26
Analysis Results
Results processing
I More than 600 experimentsI HTML-based navigation on results
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 25 / 26
Analysis Results
Results processing
I More than 600 experimentsI HTML-based navigation on results
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 25 / 26
Conclusions
Conclusion
I SocketCAN-based gateway is a reliable solution for CANbus routing.
I Kernel-space solution performs better than a user-spaceone.
I There are things to avoid:I Combining CAN traffic with Ethernet traffic.I Use of too many filters.
I The gateway will be merged in 3.2 kernel.
I Nice way of measuring Linux kernel temporal propertiesfrom outside.
I https://rtime.felk.cvut.cz/can/benchmark/2.1/
I https://rtime.felk.cvut.cz/gitweb/can-benchmark.git
Questions?
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 26 / 26
Conclusions
Conclusion
I SocketCAN-based gateway is a reliable solution for CANbus routing.
I Kernel-space solution performs better than a user-spaceone.
I There are things to avoid:I Combining CAN traffic with Ethernet traffic.I Use of too many filters.
I The gateway will be merged in 3.2 kernel.
I Nice way of measuring Linux kernel temporal propertiesfrom outside.
I https://rtime.felk.cvut.cz/can/benchmark/2.1/
I https://rtime.felk.cvut.cz/gitweb/can-benchmark.git
Questions?
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 26 / 26
Conclusions
Conclusion
I SocketCAN-based gateway is a reliable solution for CANbus routing.
I Kernel-space solution performs better than a user-spaceone.
I There are things to avoid:I Combining CAN traffic with Ethernet traffic.I Use of too many filters.
I The gateway will be merged in 3.2 kernel.
I Nice way of measuring Linux kernel temporal propertiesfrom outside.
I https://rtime.felk.cvut.cz/can/benchmark/2.1/
I https://rtime.felk.cvut.cz/gitweb/can-benchmark.git
Questions?Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 26 / 26