Adaptive Overload Control for Busy Internet Servers
Matt Welsh and David CullerUSENIX Symposium on Internet Technologies and Systems (USITS)
2003
Alex CheungNov 13, 2006
ECE1747
Outline• Motivation
• Goal
• Methodology• Detection• Overload control
• Experiments
• Comments
Motivation1. Internet services becoming important to
our daily lives:• Email• News• Trading
2. Services becoming more complex• Large dynamic content
• Requires high computation and I/O• Hard to predict load requirements of requests
3. Withstand peak load that is 1000x the norm without over-provisioning
• Solve CNN’s problem on 911
Goal• Adaptive overload control scheme at
node level by maintaining:• Response time• Throughput• QoS & Availability
Methodology - Detection
1. Look at the 90th percentile response time2. Compare with threshold and decide what to do
Weaker alternatives:• 100th percentile: does not capture “shape” of response time
curve• Throughput: does not capture user perceived performance
of the system
I ask:• What makes 90th percentile so great?• Why not 95th? 80th? 70th?• No supporting micro-experiment
1 2 3 4 5 6 7 8 9 10
Requests served
Examine 90th highest response time
Methodology – Overload Control• If response time is higher than
threshold:1. Limit service rate by rejecting selected
requests• Extension: Differentiate requests with
classes/priorities levels and reject lower class/priority requests first
2. Quality/service degradation3. Back pressure
1. Queue explosion at 1st stage (they say)• Solved by rejecting requests at 1st stage
2. Breaks the loose-coupling modular design of SEDA with out-of-band notification scheme (I say)
Methodology – Overload Control4. Forward rejected request to another
“more available” server.• “more available” – server with the most of a
particular resource:• CPU, network, I/O, hard disk
• Make decision using centralized or distributed algorithm
• Reliable state migration, possibly transactional
My take:• More complex, interesting, and actually
solves CNN’s problem with a cluster of servers!
Rate Limit
SMOOTHED
Multiplicative decrease Additive increase
Just like TCP!
10 fine-tuned parameters per stage.
Rate Limit With Class/Priority
Class/priority assignment based on:• IP address, header information, HTTP cookies
I ask:• Where is the priority assignment module implemented?• Should priority assignment be a stage of its own?• Is it not shown because complicates the diagram and makes the
stage design not “clean”?• How to classify which requests are potentially “bottleneck”
requests? Application dependent?
Quality/Service Degradation• Notify application via signal to DO service
degradation.• Application does service degradation, not
SEDA
Questions:• How is the signaling implemented?
• Out of band? • Is it possible to signal previous stages in
the pipeline? Will this SEDA’s loose-coupling design?
signal
Attach image
Send response
Experiments
Experiments - Setup
• Arashi email server (realistic experiment)• Real access workload• Real email content• Admission control
• Web server benchmark• Service degradation + 1-class admission
control
Experiments – Admission Rate
Controller response time is not as fast.
Additive increaseMultiplicative decrease
Experiments – Response Time
Why?
Why?
Experiments – Massive Load Spike
Not fair! SEDA’s parameters were fine-tuned. Apache can be tuned to stay flat too.
Experiments – Service Degradation
Service degradation and admission control kick in at roughly the same time
Experiments – Service Differentiation
• Average reject rates without service differentiation:• Low-priority: 55.5%• High-priority: 57.6%
• With service differentiation:• Low-priority: 87.9% +32.4%• High-priority: 48.8% -8.8%
Question:• Why is the drop rate for high priority
request reduced so little with service differentiation? Workload dependent?
Comments
Comments• No idea on what is the controller’s
overhead• Overload control at node level is not good:
• Node level is inefficient• Late rejection
• Node level is not user friendly:• All session state is gone if you get a reject out of the
blues ← comes without warning
• Need global level overload control scheme
• Idea/concept is explained in 2.5 pages
Comments• Rejected requests:
• Instead of TCP timeout, send static page.• (Paper says) this is better• (I say) This is worst because it leads to a out-of-
memory crash down the road:• Saturated output bandwidth• Boundless queue at reject handler
• Parameters:• How to tune them? How difficult to tune?• May be tedious tuning each stage manually.• Given a 1M stage application, need to
configure all 1M stage thresholds manually?• Automated tuning with control theory?
• Methodology of adding extensions is not shown in any figures.
Comments• Experiment is not entirely realistic:
• Inter-request think time is 20ms• realistic?
• Rejected users have to re-login after 5 min:
• All state information is gone• Frustrated users
• Two drawbacks of using response time for load detection…
Comments
1. No idea which resource is the bottleneck: CPU? I/O? Network? Memory?• SEDA can only either:
• Do admission control • Reduces throughput
• Tell application to degrade overall service
Comments
CPU I/O Network Memory
Res
ourc
e ut
iliza
tion
threshold
Default admission control:
Attach image
Send response
Reject requests
OVERLOADED!!!
… and piss off some users.
Comments
CPU I/O Network Memory
Res
ourc
e ut
iliza
tion
threshold
Service degradation WITH bottleneck intelligence:
Network is the bottleneck, so expend some CPU and memory to reduce fidelity and size of images to reduce bandwidth consumption WITHOUT reducing admission rate.
Comments2. The response time index is lagging by at
least the magnitude of the response time• 50 requests come in all at once• nreq = 100• timeout = 10s• target = 20s• Processing time per request = 1s
• Detects overload after 30s
Solution:• Compare enqueue VS dequeue rate• Overload occurs when enqueue rate >
dequeue rate• Detects overload after 10s
Questions?