MultiPath TCP - ArchitectureMultiPath TCP - Receive-Buffer
MultiPath TCP - Congestion ControlMultiPath TCP - Live-Demo
MultiPath TCP: From Theory toPractice
Sebastien BarreChristoph PaaschOlivier Bonaventure
9 mai 2011
http ://inl.info.ucl.ac.be/mptcp/
Sebastien Barre - Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 1 / 17
MultiPath TCP - ArchitectureMultiPath TCP - Receive-Buffer
MultiPath TCP - Congestion ControlMultiPath TCP - Live-Demo
MultiPath TCP
Mobile devices can connect to the Internet via different interfaces
3G
WiFi
Internet
Data-centers have a large redundant infrastructure
Dual-Homed Servers
Sebastien Barre - Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 2 / 17
MultiPath TCP - ArchitectureMultiPath TCP - Receive-Buffer
MultiPath TCP - Congestion ControlMultiPath TCP - Live-Demo
MultiPath TCP
Mobile devices can connect to the Internet via different interfaces
3G
WiFi
Internet
Data-centers have a large redundant infrastructure
Dual-Homed Servers
Sebastien Barre - Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 2 / 17
MultiPath TCP - ArchitectureMultiPath TCP - Receive-Buffer
MultiPath TCP - Congestion ControlMultiPath TCP - Live-Demo
MultiPath TCP
State of the Art
TCP is used for 95% of the Internet communications
A single TCP connection cannot be used across different interfaces.
MultiPath TCP (short MPTCP)
MPTCP allows a single data-connection to use several interfacessimultaneously.
Allows failover from one interface to another (e.g., mobile client).
Increases the bandwidth due to resource pooling.
Sebastien Barre - Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 3 / 17
MultiPath TCP - ArchitectureMultiPath TCP - Receive-Buffer
MultiPath TCP - Congestion ControlMultiPath TCP - Live-Demo
MultiPath TCP : From Theory To PracticeS. Barre, C. Paasch, O. Bonaventure
We implemented MultiPath TCP in the Linux Kernel.
Solved several design challenges that needed to be considered.
Evaluated and measured the impact of our design choices in atestbed.
Live-Demo of MPTCP at the end of this presentation.
Sebastien Barre - Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 4 / 17
MultiPath TCP - ArchitectureMultiPath TCP - Receive-Buffer
MultiPath TCP - Congestion ControlMultiPath TCP - Live-Demo
MultiPath TCP - Architecture - From Theory
standard Socket API
Transport Layer
MultiPath TCP
TCPsubflow
TCPsubflow
TCPsubflow
Application Layer
Network Layer
WiFi Wired3G
Sebastien Barre - Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 5 / 17
MultiPath TCP - ArchitectureMultiPath TCP - Receive-Buffer
MultiPath TCP - Congestion ControlMultiPath TCP - Live-Demo
MultiPath TCP - Architecture - To Practice
Sending packets over MultiPath TCP
standard Socket API
Transport Layer
Application Layer
Network Layer
MultiPath TCPMeta-socket
send-queue
TCPsubflow
Mastersubsocket
TCPsubflow
Slavesubsocket
TCPsubflow
Slavesubsocket
Sebastien Barre - Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 6 / 17
MultiPath TCP - ArchitectureMultiPath TCP - Receive-Buffer
MultiPath TCP - Congestion ControlMultiPath TCP - Live-Demo
MultiPath TCP - Architecture - To Practice
Sending packets over MultiPath TCP
standard Socket API
Transport Layer
Application Layer
Network Layer
MultiPath TCPMeta-socket
send-queue
TCPsubflow
Mastersubsocket
TCPsubflow
Slavesubsocket
TCPsubflow
Slavesubsocket
MultiPath TCP Scheduler
Sebastien Barre - Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 7 / 17
MultiPath TCP - ArchitectureMultiPath TCP - Receive-Buffer
MultiPath TCP - Congestion ControlMultiPath TCP - Live-Demo
MultiPath TCP - Architecture - To Practice
Receiving packets over MultiPath TCP
Packets can be reordered at the data-level due to delay-differences.
MPTCP
MPTCP123
4567out-of-order queue
receive-queue
subflow 1
subflow 2
subflow 1
subflow 2
low-delay path
high-delay path
8
Sebastien Barre - Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 8 / 17
MultiPath TCP - ArchitectureMultiPath TCP - Receive-Buffer
MultiPath TCP - Congestion ControlMultiPath TCP - Live-Demo
MultiPath TCP - Architecture - To Practice
Receiving packets over MultiPath TCP
A loss at the subflow-level (or network-reordering) can also causereordering at the subflow-level
MPTCP
MPTCP123
4567out-of-order queue
receive-queue
subflow 1
subflow 2
subflow 1
subflow 2
low-delay path
high-delay path
8109
dropped packet
Sebastien Barre - Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 9 / 17
MultiPath TCP - ArchitectureMultiPath TCP - Receive-Buffer
MultiPath TCP - Congestion ControlMultiPath TCP - Live-Demo
MultiPath TCP - Architecture - To Practice
Receiving packets over MultiPath TCP
Subflow-level out-of-order queues are necessary to handle theretransmission at the subflow-level
MPTCP
MPTCP123
4567out-of-order queue
receive-queue
subflow 1
subflow 2
subflow 1
subflow 2
low-delay path
high-delay path
89
retransmitted packet
10
subflowout-of-order queue
Sebastien Barre - Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 10 / 17
MultiPath TCP - ArchitectureMultiPath TCP - Receive-Buffer
MultiPath TCP - Congestion ControlMultiPath TCP - Live-Demo
MultiPath TCP - Architecture - To Practice
Receiving packets over MultiPath TCP
standard Socket API
Transport Layer
Application Layer
Network Layer
MultiPath TCPMeta-socketreceive-queueofo-queuesend-queue
TCPsubflow
Mastersubsocket
ofo-queue
TCPsubflow
Slavesubsocket
ofo-queue
TCPsubflow
Slavesubsocket
ofo-queue
Sebastien Barre - Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 11 / 17
MultiPath TCP - ArchitectureMultiPath TCP - Receive-Buffer
MultiPath TCP - Congestion ControlMultiPath TCP - Live-Demo
MultiPath TCP - Architecture - To Practice
Interconnected testbed with two separate paths at 1Gbps
1Gbps
1Gbps
0
500
1000
1500
2000
1000 2000 3000 4000 5000 6000 7000 8000 9000
iperfg
oodp
ut(M
bps)
MSS (bytes )
Sebastien Barre - Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 12 / 17
MultiPath TCP - ArchitectureMultiPath TCP - Receive-Buffer
MultiPath TCP - Congestion ControlMultiPath TCP - Live-Demo
MultiPath TCP - Architecture - To Practice
regular TCP can only use one single path
1Gbps
1Gbps
regular TCP
0
500
1000
1500
2000
1000 2000 3000 4000 5000 6000 7000 8000 9000
iper
fgoo
dput
(Mbp
s)
MSS (bytes )
regular TCP
Sebastien Barre - Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 12 / 17
MultiPath TCP - ArchitectureMultiPath TCP - Receive-Buffer
MultiPath TCP - Congestion ControlMultiPath TCP - Live-Demo
MultiPath TCP - Architecture - To Practice
MultiPath TCP uses both paths simultaneously
1Gbps
1Gbps
MPTCP
0
500
1000
1500
2000
1000 2000 3000 4000 5000 6000 7000 8000 9000
iper
fgoo
dput
(Mbp
s)
MSS (bytes )
MultiPath TCPregular TCP
Sebastien Barre - Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 12 / 17
MultiPath TCP - ArchitectureMultiPath TCP - Receive-Buffer
MultiPath TCP - Congestion ControlMultiPath TCP - Live-Demo
MultiPath TCP - Receive-Buffer - From Theory
Regular TCP : receive-buffer should be twice the BDP
Support a fast-retransmitSupport network-level reordering
MultiPath TCP : Higher reordering possible due todelay-differences of the paths.
MPTCP
MPTCP123
4567out-of-order queue
receive-queue
subflow 1
subflow 2
subflow 1
subflow 2
low-delay path
high-delay path
8
Sebastien Barre - Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 13 / 17
MultiPath TCP - ArchitectureMultiPath TCP - Receive-Buffer
MultiPath TCP - Congestion ControlMultiPath TCP - Live-Demo
MultiPath TCP - Receive-Buffer - To Practice
⇒A subflow on a slow path (high RTT) may block a subflow on a fastpath from transmitting.
RTTmax
Sebastien Barre - Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 14 / 17
MultiPath TCP - ArchitectureMultiPath TCP - Receive-Buffer
MultiPath TCP - Congestion ControlMultiPath TCP - Live-Demo
MultiPath TCP - Receive-Buffer - To Practice
⇒A subflow on a slow path (high RTT) may block a subflow on a fastpath from transmitting.
⇒We need to cope with one fast-retransmit and network-level reorderingon this slow path
RTTmax ∗ 2
Sebastien Barre - Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 14 / 17
MultiPath TCP - ArchitectureMultiPath TCP - Receive-Buffer
MultiPath TCP - Congestion ControlMultiPath TCP - Live-Demo
MultiPath TCP - Receive-Buffer - To Practice
⇒A subflow on a slow path (high RTT) may block a subflow on a fastpath from transmitting.
⇒We need to cope with one fast-retransmit and network-level reorderingon this slow path
⇒During this time, all other subflows should be able to transmit at fullspeed (BWi = bandwidth of subflow i)
RTTmax ∗ 2 ∗∑
i∈subflows
BWi
Sebastien Barre - Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 14 / 17
MultiPath TCP - ArchitectureMultiPath TCP - Receive-Buffer
MultiPath TCP - Congestion ControlMultiPath TCP - Live-Demo
MultiPath TCP - Receive-Buffer - To Practice
Regular TCP only gets 100Mbps of goodput
100Mbps
100Mbps+ 0ms
regular TCP
40
60
80
100
120
140
160
180
200
2 4 6 8 10 12 14 16
iperfg
oodp
ut(M
bps)
rcvbuf (MB)
TCP (0 ms ec)
Sebastien Barre - Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 15 / 17
MultiPath TCP - ArchitectureMultiPath TCP - Receive-Buffer
MultiPath TCP - Congestion ControlMultiPath TCP - Live-Demo
MultiPath TCP - Receive-Buffer - To Practice
MultiPath TCP doubles the goodput
100Mbps
100Mbps+ 0ms
MPTCP
40
60
80
100
120
140
160
180
200
2 4 6 8 10 12 14 16
iperfg
oodp
ut(M
bps)
rcvbuf (MB)
MPTCP (0 ms ec , 0 ms ec)
TCP (0 ms ec)
Sebastien Barre - Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 15 / 17
MultiPath TCP - ArchitectureMultiPath TCP - Receive-Buffer
MultiPath TCP - Congestion ControlMultiPath TCP - Live-Demo
MultiPath TCP - Receive-Buffer - To Practice
10ms and 100ms of difference has not a huge influence
100Mbps
100Mbps+ 10ms
MPTCP
40
60
80
100
120
140
160
180
200
2 4 6 8 10 12 14 16
iperfg
oodp
ut(M
bps)
rcvbuf (MB)
MPTCP (0 ms ec , 0 ms ec)MPTCP (0 ms ec , 10 ms ec)
TCP (0 ms ec)
Sebastien Barre - Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 15 / 17
MultiPath TCP - ArchitectureMultiPath TCP - Receive-Buffer
MultiPath TCP - Congestion ControlMultiPath TCP - Live-Demo
MultiPath TCP - Receive-Buffer - To Practice
10ms and 100ms of difference has not a huge influence
100Mbps
100Mbps+ 100ms
MPTCP
40
60
80
100
120
140
160
180
200
2 4 6 8 10 12 14 16
iperfg
oodp
ut(M
bps)
rcvbuf (MB)
MPTCP (0 ms ec , 0 ms ec)MPTCP (0 ms ec , 10 ms ec)
MPTCP (0 ms ec , 100 ms ec)
TCP (0 ms ec)
Sebastien Barre - Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 15 / 17
MultiPath TCP - ArchitectureMultiPath TCP - Receive-Buffer
MultiPath TCP - Congestion ControlMultiPath TCP - Live-Demo
MultiPath TCP - Receive-Buffer - To Practice
500ms of difference affects the goodput significantly
100Mbps
100Mbps+ 500ms
MPTCP
40
60
80
100
120
140
160
180
200
2 4 6 8 10 12 14 16
iperfg
oodp
ut(M
bps)
rcvbuf (MB)
MPTCP (0 ms ec , 0 ms ec)MPTCP (0 ms ec , 10 ms ec)
MPTCP (0 ms ec , 100 ms ec)MPTCP (0 ms ec , 500 ms ec)
TCP (0 ms ec)
Sebastien Barre - Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 15 / 17
MultiPath TCP - ArchitectureMultiPath TCP - Receive-Buffer
MultiPath TCP - Congestion ControlMultiPath TCP - Live-Demo
MultiPath TCP - Congestion Control
Several subflows will ”eat up” regular TCP
33%
66%MPTCP
2 subflows
regular TCP shared Bottleneck
1 2 3Number of MPTCP-Subflows
0
20
40
60
80
100
Aver
age
Good
put i
n M
bps
TCPMPTCP-Reno
Sebastien Barre - Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 16 / 17
MultiPath TCP - ArchitectureMultiPath TCP - Receive-Buffer
MultiPath TCP - Congestion ControlMultiPath TCP - Live-Demo
MultiPath TCP - Congestion Control
Our implementation of the Coupled Congestion Control 1 is fair to regularTCP across shared bottlenecks and avoids the paths with congested links.
50%
50%MPTCP
2 subflows
regular TCP shared Bottleneck
1 2 3Number of MPTCP-Subflows
0
20
40
60
80
100
Aver
age
Good
put i
n M
bps
TCPMPTCP-CCC
1. D. Wischik, C. Raiciu, A. Greenhalgh, M. Handley. ”Design, implementation andevaluation of congestion control for multipath TCP”. NSDI’2011
Sebastien Barre - Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 16 / 17
MultiPath TCP - ArchitectureMultiPath TCP - Receive-Buffer
MultiPath TCP - Congestion ControlMultiPath TCP - Live-Demo
MultiPath TCP - Live-Demo
Access http ://inl.info.ucl.ac.be/mptcp to download ourimplementation (source-code, Live-CD,...).
Contributions are always welcome ! ! !
Setup for the Live-Demo :
Sebastien Barre - Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 17 / 17
MultiPath TCP - ArchitectureMultiPath TCP - Receive-Buffer
MultiPath TCP - Congestion ControlMultiPath TCP - Live-Demo
MultiPath TCP - Congestion Control - Theory
Coupled Congestion Control 2
For each ack on subflow i :
cwndi = cwndi + min(α/cwndtot , 1/cwndi )
α = cwndtot
maxi (cwndi∗mss2
i
RTT 2i
)
(∑
icwndi∗mssi
RTTi)2
2. D. Wischik, C. Raiciu, A. Greenhalgh, M. Handley. ”Design, implementation andevaluation of congestion control for multipath TCP”. NSDI’2011
Sebastien Barre - Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 17 / 17
MultiPath TCP - ArchitectureMultiPath TCP - Receive-Buffer
MultiPath TCP - Congestion ControlMultiPath TCP - Live-Demo
MultiPath TCP - Congestion Control - Practice
Congestion Control in the Linux Kernel
Limitations :
Congestion window is expressed in packets
Linux Kernel cannot handle floating point numbers
Coupled Congestion Control implementation
Count the number of acknowledged packets per subflow incwnd cnti .
Increase congestion window by one, ifcwnd cnti > max(totcwnd/α, 1/cwndi )
Calculate alpha by minimizing the number of divisions and scalingthe remaining ones :
α = cwndtotcwndmax ∗ scalenum
(∑
irttmax∗cwndi∗scaleden
rtti)2
Sebastien Barre - Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 17 / 17