Deploying Safe User-Level Network Services withicTCP
Haryadi S. Gunawi Andrea C. Arpaci-Dusseau
Remzi H. Arpaci-Dusseau
The ADvanced Systems Laboratory (ADSL) Univ. of Wisconsin - Madison
Motivation Vast number of proposed modifications to TCP/IP
Some adopted, others not deployed TCP Vegas [SIGCOMM `94] TCP Nice [OSDI `02] Congestion Manager (CM) [SIGCOMM `99]
Heart of deployment problem: OS kernel OS tend to be substantial and complex Vendors dislike changing them when benefit is not imminent A range of OS-es must implement the modifications
Transition of good research ideas into wide-spread use is slow
Emerging different network environments Wireless (lossy network), Load-balancing (packet reordering) Different extensions for different network environment TCP only support some
Current TCP implementation
TCP
TCP Applications
Reno
f ( )
f ( )
in-kernel extensions
Inside to outside
TCP
TCP Applications
f ( )
Information Control
Why in-kernel extensions?
Information and control only available within the kernel
Ex: TCP Vegas• Information: per-packet
RTT and TCP States• Control: Congestion
Window Questions?
Can extensions be built in application layer?
What information and control need to be exposed?
Proposed Framework icTCP (“information and control” TCP)
Address the problem of deployment Slightly modified in-kernel TCP stack Exports key pieces of information Provides safe control to user-level libraries Given information and control, extensions can be built in
application layer Evaluation
Converting TCP to icTCP requires a small amount of additional code
The resulting flow is TCP friendly Minimal overhead Implement 6 user-level extensions Little complexity in implementing extensions at user-
level
Benefits
lib-f ( )
icTCP
Apps
Deployable
New services packaged as libraries and easily downloaded.
lib-f icTCP
App1 App2
lib-g
Flexible
Developers can tailor libraries to the their needs
lib-g ( )
icTCP
Apps
lib-f ( )
Composable
Services are used together for more powerful functionality
icTCP facilitates the development of user-level extensions
Outline Motivation and Overview icTCP Design and Implementation
Information Control
5 Axes of Evaluation Conclusion
icTCP Design Different TCP
connections, different libraries
User-Libraries on top of icTCP
icTCP exposes Information and Control
Information Which information should
be exposed? Too little limited extensions Too much expanded
interface
2 types of Information: Variables part of TCP
specification (RFC 793)• cwnd, ssthresh
Message list and Ack list• More detailed information• History of recent packets and
acknowledgements• Enabled by user-level services
to save memory
TCP Clients
User Libraries
How and when to obtain information? Polling
BSD socket interfaces Unnecessary run-time
overhead• Kernel-user space copy
When to obtain accurately?
TCP variables are updated:
• upon receipt of an ACK• end of a round
Interrupt mechanism icTCP notifies application
when these events happen
CPU Overhead
0
2
4
6
8
10
Reno Poll /Send()
Poll /Ack
Poll /Round
CPU
Utilz
atio
n (%
)
Control Allow internal TCP variables
to be externally set in a safe manner
What variables and what values?
TCP-Friendly: don’t harm other competing standard flows
Philosophy: icTCP must be conservative
Control is allowed if not aggressive
Control 10 (virtual) variables that can be safely set
User Libraries
TCP Clients
Implementation of Safe-Control Idea: Virtual variables
congestion window (cwnd) virtual congestion window (vcwnd) Manipulate policies through virtual variables TCP Reno keeps running using the original variables
Safe Ranges Enforce friendliness 0 ≤ vcwnd ≤ cwnd Without safe ranges, resulting flows not TCP-friendly
Choosing 10 variables, defining safe-ranges Check safe-setting theoretically and empirically Theoretically: Reno equation
• congestion window: control how many packets in the network• vcwnd > cwnd : more packets in the network, thus more aggressive
Empirical Prove• Set virtual congestion window outside the safe ranges
Outline Motivation and Overview icTCP Design and Implementation Framework Evaluation Conclusion
5 Axes of Evaluation
12
4 3
5
How easy to convert TCP to icTCP?
Is icTCP friendly?
What are the overheads? Does it scale?
What types of extension can be built and deployed with icTCP?
How difficult to develop TCP extensions in this way?
Converting TCP to icTCP icTCP in Linux 2.4.18 Add 316 lines of code
Non-intrusive modifications
TCP toicTCP1
Network Safety: Unconstrained icTCP
2Friendly?
TunconsReno
Reno
RenoTReno
TReno
Runcons :Tuncons
TCP-Friendly: ~ 1
UnconstrainedicTCP
(allow variable setting to any value)
TCP-Unfriendly
Network Safety: Constrained icTCP
TconsReno
TReno
Rcons :Tcons
Reno
RenoTReno
Safety Safe Ranges are required!
TCP-Friendly
2Friendly?
UnconstrainedicTCP
ConstrainedicTCP
.
(allow variable setting within the safe ranges)
TCP-Unfriendly
Scalability and Overhead What is the overhead? Does it scale?
Rate of getting info and setting variables Different extensions, get/set at different rate
Two factors: Per-ack or per-round interrupts Need the message list and ack list
3 synthetic libraries: per-ack interrupt per-round interrupt per-round interrupt + gets message list
3 Scale?Overhead?
OverheadScale?Overhead?3
per-ack (4 conns)
(96 conns)
per-round 12%
per-ack 30%
S
R1
R3
R2
R4
12 MB/s per-round + msgList (8 conns)
Noticeable overhead, but not prohibitive
TCP Extensions Range of TCP extensions can be built on top of icTCP Implement 3 sets case studies (6 extensions):
Congestion window:• TCP Vegas [Brakmo, SIGCOMM ’94]• TCP Nice [Venkataramani, OSDI ’02]• Congestion Manager (CM) [Balakhrisnan, SIGCOMM ’99]
Duplicate threshold• Reordering-Robust Ext [Zhang, ICNP ‘03] • Efficient Fast Retransmit [Tamura, LCN ‘98]
Retransmission Timeout• Eifel RTO [CCR ‘00]
4Range ofExtensions?
icTCP
lib-Vegas
User-level TCP Vegas Implementation
TCP Clients
TCP states
msg list
Vegas
f vcwnd
vcwnd
4Range ofExtensions?
TCP Vegas [Brakmo et.al.,SIGCOMM ’94]
per-round interrupt
Using information and control is simple Complexity is within the algo itself
Does it work?
User-Level Lib. TCP VegasIn-kernel TCP Vegas
Reno
Same Behavior
TCP Reno: Send more packets until drop.
TCP Vegas: Maintain the “right” amount of extra data in the network
e.g. keep only 3 packets in the bottleneck queue
4Range ofExtensions?
icTCP Strengths Implement less aggressive TCP
variants at user-level No need to push changes into the
kernel Ideally suited for tuning parameters
whose optimal values depend upon the environment
Lossy Network retransmit faster Use Efficient Fast Retransmit
Packet reordering postpone retransmission Reordering Robust (RR) Extensions
Different environment, opposing solutions
TCP favors one solution Correcting errors in parameter values
Eifel RTO [CCR ’00]: RTO prediction flaw when RTT drops
In newer Linux version, this prediction flaw is corrected with adding 2 new lines of code
Must wait vendors to correct this flaw
4Range ofExtensions?
TCP + f lib-f icTCP
App2
RR
PacketReordering
dt > 3dt < 3
App1
Lossy Network
EFR
Other approaches Can’t implement all extensions
Only allow 10 existing variables to be set Conservative: Safety
STP (Self-Spreading Transport Protocols) [SOSP ’03]
Remotely upgrade other’s protocol stack More TCP extensions can be deployed Use separate module to enforce
friendliness
Why we don’t use enforcer Drawback: terminate non-conforming
flows icTCP modulates aggressive flows
4Range ofExtensions?
Set existing
variables with limits
lib-f icTCP
Apps
Set existingvars to any
values
lib-f icTCP
Apps
Enforcer
vs.
Ease of Development How difficult to develop TCP extensions in
this way? Complexity at user-level vs. in-kernel
5DifficultDevelop?
Libray vs In-Kernel Implementation
0
200
400
600
TCP Vegas TCP Nice CM RR
Extensions
Line
s of
Cod
e
User-Level Library In-Kernel
1200(*)
438
Composability Composing multiple libraries
Vegas: good for small bottleneck queues RR: good for reordering Environment where both situations exist?
• Vegas+RR: Better throughput
TCP Clients
icTCP
Vegas RR RR
Vegas
5DifficultDevelop?
Conclusion icTCP
Philosophy: Information and Control Non-intrusive, simple, and TCP-friendly Systems with icTCP reap benefits of
user-level TCP extensions
Change TCP once now, no need to change it later
Questions?
The ADvanced Systems Laboratory
www.cs.wisc.edu/adsl
Related Work Mogul et.al. [HotNets-2003]
Arbitrary state setting Web100 and Net100
Export a variety of per-connection statistics Does not ensure network safety
STP [SOSP-2003] Remotely upgrade other’s protocol stack Invasive changes to the kernel Additional machinery to ensure friendliness
InfoTCP/infokernel [SOSP-2003] icTCP exposes more information icTCP allows services to directly set cwnd inside of TCP icTCP allows more variables to be controlled