© author(s) of these slides including research results from the KOM research network and TU Darmstadt; otherwise it is specified at the respective slide Template all v.3.4
A Programming Model for Application-
defined Multipath TCP Scheduling
ACM/IFIP/USENIX Middleware 2017
Alexander Frömmgen
Alexander Frömmgen Amr Rizk
Tobias Erbshäußer Max Weller
Boris Koldehofe Alejandro Buchmann
Ralf Steinmetz
Application
Programmable
Scheduler
Userland
Kernel
Network Stack
Sending Queue Subflows
Send Data
Load MPTCP Scheduler
KOM – Multimedia Communications Lab
Multipath TCP in a Nutshell
Server Sending Queue
p5
Scheduler
Subflow 2 LTE
Subflow 1 WiFi
p6
Receive Queue
p1
Application
p2
Client
0x01
KOM – Multimedia Communications Lab
Multipath TCP in a Nutshell
Server Sending Queue
p5
Scheduler
Subflow 2 LTE
Subflow 1 WiFi
p6
Receive Queue
p1
Application
p2
Client
MPTCP Packet
Trigger Subflow Establishment
Path Management
Control Information, e.g., Data Sequence Numbers
TCP Options for MPTCP
Fairness on Joint Paths
Congestion Control
Map Packets on Subflows
Scheduler
1 2
3 4
0x01
KOM – Multimedia Communications Lab
Multipath TCP Scheduling
Server
Scheduler
Client
Packet 2
Subflow 2 LTE
Subflow 1 WiFi
Packet 1
0x02
RTT = 40ms
BW = …
P_IN_FLY = 0
CWND = 10
RTT = 10ms
BW = …
P_IN_FLY = 0
CWND = 10
Images: https://www.youtube.com/watch?v=U2T1_VrTcoo
KOM – Multimedia Communications Lab
Multipath TCP Scheduling
Server
Scheduler
Client Subflow 2 LTE
Subflow 1 WiFi
Packet 2
Packet 1
0x02
RTT = 40ms
BW = …
P_IN_FLY = 0
CWND = 10
RTT = 10ms
BW = …
P_IN_FLY = 0
CWND = 10
Schedule on Subflow with minimum round-trip time (RTT). Intuition:
Images: https://www.youtube.com/watch?v=U2T1_VrTcoo
KOM – Multimedia Communications Lab
Multipath TCP Scheduling
Server
Scheduler
Client Subflow 2 LTE
Subflow 1 WiFi
Packet 5 Packet 4
0x02
RTT = 40ms
BW = …
P_IN_FLY = 0
CWND = 10
Images: https://www.youtube.com/watch?v=U2T1_VrTcoo
Schedule on Subflow with minimum round-trip time (RTT).
Packet 1
RTT = 10ms
BW = …
P_IN_FLY = 10
CWND = 10
Packet 11
KOM – Multimedia Communications Lab
Multipath TCP Scheduling
Server
Scheduler
Client Subflow 2 LTE
Subflow 1 WiFi
Packet 5 Packet 4
0x02
RTT = 40ms
BW = …
P_IN_FLY = 0
CWND = 10
Images: https://www.youtube.com/watch?v=U2T1_VrTcoo
Packet 1
RTT = 10ms
BW = …
P_IN_FLY = 10
CWND = 10
Packet 11
Schedule on Subflow with minimum round-trip time (RTT).
which is not saturated (Congestion window larger than packets in flight).
KOM – Multimedia Communications Lab
Multipath TCP Scheduling
Motivating Example
Home Network WiFi and LTE Germany
Constant Bitrate Stream
0x03
Server Sending Queue
p5
Scheduler
Subflow 2 LTE
Subflow 1 WiFi
p6
Receive Queue
p1
Application
p2
Client ~RTT 40ms
~RTT 10ms
KOM – Multimedia Communications Lab
Multipath TCP Scheduling
Motivating Example
Home Network WiFi and LTE Germany
Constant Bitrate Stream
0x03
Server Sending Queue
p5
Scheduler
Subflow 2 LTE
Subflow 1 WiFi
p6
Receive Queue
p1
Application
p2
Client ~RTT 40ms
~RTT 10ms
KOM – Multimedia Communications Lab
Multipath TCP Scheduler Overview
Name Domain Pref. Available in Linux
Implementation
Min RTT (default) General Purpose Binary
Round-Robin Academic Binary
Redundant Thin Flows Binary
Compensate Loss Short Datacenter Flows ? No
Video and Energy ? No
DASH Video Yes No
...
0x04
KOM – Multimedia Communications Lab
Multipath TCP Scheduler Overview
Great Concepts without MPTCP Implementation
H. Xu and B. Li. RepFlow: Minimizing flow completion times with replicated flows in data centers.
In IEEE INFOCOM, pages 1581–1589, 2014
X. Corbillon, R. Aparicio-Pardo, N. Kuhn, G. Texier, and G. Simon. Cross-layer scheduler for video streaming over MPTCP. In ACM MMSys, 2016
N. Kuhn, E. Lochin, A. Mifdaoui, G. Sarwar, O. Mehani, and R. Boreli. DAPS: intelligent delay-aware packet scheduling for
multipath transport. In IEEE ICC, 2014.
KOM – Multimedia Communications Lab
Research Questions
How can we systematically specify and execute MPTCP schedulers?
How can we enable application-defined MPTCP scheduling?
0x06
KOM – Multimedia Communications Lab
Part I: Towards a Programmable Scheduler in
the Network Stack
Application
Programmable Scheduler
Userland
Kernel
Network Stack
Sending Queue Subflows
Send Data
Load MPTCP Scheduler
0x07
KOM – Multimedia Communications Lab
Model of the Scheduler Environment
Sch
ed
ule
r
Sending Queues Subflows
PUSH
op*
*operation, e.g., POP
PUSH
Application
Userland
Kernel
0x08
KOM – Multimedia Communications Lab
Model of the Scheduler Environment
Subflow sbf1
Queue Qsbf1
Sending Queue Q
Reinjection Queue RQ Sch
ed
ule
r
Prop.: rtt, cwnd, …
Subflow sbf2
Queue Qsbf2
Prop.: rtt, cwnd, …
p2 p3 p4
p1
Packets in Flight QU
p1
Sending Queues Subflows
PUSH
PUSH
op*
op*
*operation, e.g., POP
PUSH
op*
Application
Userland
Kernel
0x08
KOM – Multimedia Communications Lab
Specifying Multipath TCP Schedulers
Expressiveness
Isolation
Timely Execution
1. Modelled Elements as Entities: Set of Subflows with their properties, Queues of Packets with their Properties 2. Declarative Packet and Subflow Selection (Filter, Min, Max) 3. No Recursion, No Functions, Limited Loops 4. Variables with Single Assignment, Implicit and Static Type System 5. No, One, or Multiple Packets per Scheduler Execution
Design Decisions
Requirements
0x09
KOM – Multimedia Communications Lab
0000000000001010
Systematically Specify MPTCP Schedulers
Domain Specific Specification Language
1 VAR sbfCandis = SUBFLOWS.FILTER(
2 sbf => sbf.CWND > sbf.SKBS_IN_FLIGHT + sbf.QUEUED
3 AND !sbf.TSQ_THROTTLED AND !sbf.LOSSY);
4
5 VAR backSbf = sbfCandis.FILTER(
6 sbf => sbf.IS_BACKUP).MIN(sbf => sbf.RTT);
7 VAR nonBackSbf = sbfCandis.FILTER(
8 sbf => !sbf.IS_BACKUP).MIN(sbf => sbf.RTT);
9
10 IF (nonBackSbf.RTT_MS > 100 AND backSbf.RTT_MS < 80) {
11 backSbf.PUSH(Q.POP());
12 } ELSE {
13 nonBackSbf.PUSH(Q.POP());
14 }
Example: Preference-aware RTT-sensitive Scheduler
0x0A
KOM – Multimedia Communications Lab
0000000000001010
Systematically Specify MPTCP Schedulers
Domain Specific Specification Language
1 VAR sbfCandis = SUBFLOWS.FILTER(
2 sbf => sbf.CWND > sbf.SKBS_IN_FLIGHT + sbf.QUEUED
3 AND !sbf.TSQ_THROTTLED AND !sbf.LOSSY);
4
5 VAR backSbf = sbfCandis.FILTER(
6 sbf => sbf.IS_BACKUP).MIN(sbf => sbf.RTT);
7 VAR nonBackSbf = sbfCandis.FILTER(
8 sbf => !sbf.IS_BACKUP).MIN(sbf => sbf.RTT);
9
10 IF (nonBackSbf.RTT_MS > R1 AND backSbf.RTT_MS < R2) {
11 backSbf.PUSH(Q.POP());
12 } ELSE {
13 nonBackSbf.PUSH(Q.POP());
14 }
Example: Preference-aware RTT-sensitive Scheduler
0x0A
KOM – Multimedia Communications Lab
Systematically Specify and
Execute MPTCP Schedulers
Specified schedulers are executable in the Linux Kernel
ProgMP
Lexer
Parser
Kernel
Control Flow Graph Scheduler
Optimizer
Native Code
Userland Write to /proc/…/schedulers
Interpreter
eBPF Compiler
Compiler Frontend
Network Stack
Uses
Ahead of Time Compiler
0x0B
KOM – Multimedia Communications Lab
Application-aware Scheduling in
Multi-Tenancy Cloud Environments
Application, e.g., Webserver
Ext. API Socket API
Isolated Scheduler
1
2
3
4
5
6
7
8
9
10
11
12
13
import socket
from progmp import ProgMp
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("10.0.0.2", 8080))
try:
ProgMp.loadScheduler("python_api_example...")
ProgMp.setScheduler(s, "python_api_example")
except:
print "Scheduler loading error."
ProgMp.setRegister(s, Progmp.R1(), 50)
s.send("Multipath is awesome!")
0x0C
KOM – Multimedia Communications Lab
Abstraction vs. Overhead
0x0D
KOM – Multimedia Communications Lab
Abstraction vs. Overhead
The runtime environment induces a small overhead, which is acceptable for most application scenarios.
0x0D
KOM – Multimedia Communications Lab
Part II:
Design of Novel Multipath TCP Schedulers
0x0E
Round-trip Time-aware
Constant Bitrate Stream Scheduling
Redundant Scheduling
HTTP-aware Scheduling
More in Paper and Under Review*
* Multipath TCP Scheduling for Thin Streams: Active Probing and One-way Delay-awarness by Alexander Frömmgen,
Jens Heuschkel and Boris Koldehofe, IEE ICC 2018 (to appear).
KOM – Multimedia Communications Lab
A Close Look at Redundant Schedulers
Seq 2
Seq 3 Seq 4
Seq 5
Q
Subflow 1 (RTT 50ms)
In Flight
Seq 3 Subflow 2 (RTT 150ms)
Receiver Sender
Ack 2
Seq 2
Seq 5
Seq 4
Seq 7
Seq 6
Seq 3
Should we send the fresh packet or the old packet when the acknowledgement arrives at the sender? Ack 2
Seq 6 Seq 5
0x0F
KOM – Multimedia Communications Lab
A Close Look at Redundant Schedulers
Seq 2
Seq 3 Seq 4
Seq 5
Q
Subflow 1 (RTT 50ms)
In Flight
Seq 3 Subflow 2 (RTT 150ms)
Receiver Sender
Ack 2
Seq 2
Seq 5
Seq 4
Seq 7
Seq 6
Seq 3
Should we send the fresh packet or the old packet when the acknowledgment arrives at the sender? Ack 2
Seq 6 Seq 5
1
2
3
4
5
6
7
8
FOREACH(VAR sbf IN sbfCandidates) {
VAR skb = QU.FILTER(s => !s.SENT_ON(sbf).TOP;
IF(skb != NULL) {
sbf.PUSH(skb);
} ELSE {
sbf.PUSH(Q.POP());
}
}
1
2
3
4
5
6
IF(!sbfCandidates.EMPTY) {
FOREACH(VAR sbf IN sbfCandidates) {
sbf.PUSH(Q.TOP);
}
DROP(Q.POP());
}
Prefer Global Fresh Packets
Prefer Per Subflow Fresh Packets
0x10
ProgMP enables rapid specification and evaluation of schedulers.
KOM – Multimedia Communications Lab
A Close Look at Redundant Schedulers
ProgMP enables novel redundant schedulers, which outperform established approaches.
Better
0x11
KOM – Multimedia Communications Lab
Side Note: How can we systematically compare
and evaluate scheduler design decisions?
0x12
Configurations
Environ.
Single Experiment Result Add Config. and Env. Variations
Fix / Improve Implementation
Add Protocols and Algorithms
Inspect Results: Interactive Data Analysis and Exploration
Web Frontend to Manage Exp.
Exp. Study
Exp.
Config.
Env.
x
Scalable Experiment Execution
Iterative Refinments in the Research Process
Exp.
KOM – Multimedia Communications Lab
Side Note: How can we systematically compare
and evaluate scheduler design decisions?
0x12
Configurations
Environ.
Single Experiment Result Add Config. and Env. Variations
Fix / Improve Implementation
Add Protocols and Algorithms
Inspect Results: Interactive Data Analysis and Exploration
Web Frontend to Manage Exp.
Exp. Study
Exp.
Config.
Env.
x
Scalable Experiment Execution
Iterative Refinments in the Research Process
Exp. A Framework for the Management, the Scalable Execution and Interactive Analysis of Extensive Network Experiments
https://maci-research.net
KOM – Multimedia Communications Lab
HTTP/2-aware Scheduling
0x13
ProgMP enables HTTP/2-aware Scheduling.
KOM – Multimedia Communications Lab
HTTP/2-aware Scheduling
0x14
Note that the large variance is partly caused by a high variance of the underlying real world environment.
KOM – Multimedia Communications Lab
Conclusion
• Specification and execution of MPTCP schedulers • Application-defined MPTCP scheduling
The presented programming model enables:
• RTT-aware scheduler • Constant bitrate schedulers • Flavors of redundant schedulers • HTTP-aware scheduler • …
We presented the first programming model for Multipath TCP scheduling.
We proposed and evaluated sophisticated novel schedulers.
0x15
KOM – Multimedia Communications Lab
Conclusion
• Specification and execution of MPTCP schedulers • Application-defined MPTCP scheduling
The presented programming model enables:
• RTT-aware scheduler • Constant bitrate schedulers • Flavors of redundant schedulers • HTTP-aware scheduler • …
We presented the first programming model for Multipath TCP scheduling.
We proposed and evaluated sophisticated novel schedulers.
0x15
Try it! https://progmp.net
KOM – Multimedia Communications Lab
Questions
KOM – Multimedia Communications Lab http://www.tessares.net/highlights-from-advances-in-networking-part-1/
KOM – Multimedia Communications Lab
Multipath TCP Scheduler Overview
KOM – Multimedia Communications Lab
Enabling per Application-aware Scheduling in
Multi-Tenancy Cloud Environments
Queue Q
Packet Scheduler p2 p3 p4
Extended API
Choose Scheduler
Registers
Set Register Set Packet Property
proc fs
p5
PROP
Parser / Optimizer / Repository
Application, e.g., Webserver
Socket API
R1 R2 …
Python Application
Application B JRE
Ext. API Socket API
Isolated Scheduler Runtime Environment Isolated Scheduler …
Users
pace
K
ern
el