1 Accumulation-based Congestion Control Yong Xia, David Harrison † , Shivkumar Kalyanaraman, Kishore Ramachandran, Arvind Venkatesan † ECSE and CS † Departments Rensselaer Polytechnic Institute, Troy, NY 12180, USA Abstract—This paper 1 generalizes the TCP Vegas conges- tion avoidance mechanism and uses accumulation, buffered packets of a flow inside network routers, as a congestion measure based on which a family of congestion control schemes can be derived. We call this model accumulation- based congestion control (ACC), which fits into the nonlin- ear optimization framework proposed by Kelly [20]. The ACC model serves as a reference for packet-switching net- work implementations. We show that TCP Vegas is one pos- sible scheme under this model. It is well known that Vegas suffers from round trip propagation delay estimation error and reverse path congestion. We therefore design a new Monaco scheme that solves these problems by employing an out-of-band, receiver-based accumulation estimator, with the support of two FIFO priority queues from (congested) routers. Comparisons between these two schemes demon- strate that Monaco achieves better performance than Vegas and does not suffer from the problems mentioned above. We use ns-2 simulations and Linux implementation experiments to show that the static and dynamic performance of Monaco matches the theoretic results. One key issue regarding the ACC model in general, i.e., the scalability of router buffer requirements and a solution using a virtual queuing algo- rithm, are discussed and evaluated. I. I NTRODUCTION Much research has been conducted to achieve stable, efficient and fair operation of packet-switching networks. TCP congestion control [18], a set of end-to-end mecha- nisms, has been widely acknowledged for its critical role in maintaining stability of the Internet. Among them, TCP Reno [2] infers network congestion by detecting packet loss that is assumably caused only by congestion; alter- natively, TCP Vegas [10] measures backlog, the number of buffered packets inside routers along the path, to detect network congestion and demonstrates better performance than Reno. Unfortunately, Vegas has technical problems inherent to its backlog estimator that prevent it from func- tioning properly. There has been a substantial amount of 1 This work was supported by NSF under contracts ANI-9806660 and ANI-9819112 and a grant from Intel Corp. Part of the paper has been presented at the IEEE International Conference on Communications in Anchorage, Alaska, USA on May 11-15, 2003. work on this issue, such as [1] [26] [17] [8] [9] [25] [12] and references therein, which we review in Section II. But none of them provides a solution to estimate backlog un- biasedly in case of round trip propagation delay estimation error or reverse path congestion. In this paper, we offer a solution to this problem and de- velop a systematic model to generalize Vegas’ congestion avoidance mechanism. Formally, we define in a bit-by-bit fluid model the backlog (hereafter we call it accumulation) as a time-shifted, distributed sum of queue contributions of a flow at a set of FIFO routers on its path. The central idea is to control flows’ rate by controlling their accumulations in an end-to-end and distributed manner. We study a set of closed-loop congestion control schemes that are all based upon the idea of keeping a target accumulation for each flow individually. The key concepts for this accumulation-based conges- tion control (ACC) model are developed in Section III. An ACC model has two components: congestion estima- tion and congestion response. The former defines a con- gestion measure (i.e., accumulation) and provides an im- plementable estimation of the measure; while the latter defines an increase/decrease policy for the source throt- tle. A class of control algorithms, including the additive- increase/additive-decrease (AIAD) policy [11], Mo and Walrand’s proposal [27] and a proportional control, can be used. Based on previous research [27] [20], we show that the equilibria of all these algorithms achieve the same proportional fairness in the appendix. To instantiate the ACC model, choices can be made in each of the ACC components to put together the entire scheme. We describe two example schemes in Section IV. We show that Vegas congestion avoidance attempts to esti- mate accumulation and thus fits into the ACC family. But it often fails to provide an unbiased accumulation estimate. We therefore design a new scheme called Monaco that em- ulates the ACC fluid model in a better way. Particularly, Monaco solves the Vegas problem by employing an out-of- band, receiver-based accumulation estimator. We provide resolution to a number of concerns regarding accumula- tion estimation in Section IV-C. Section V demonstrates the steady state and dynamic performance of Monaco us-
Transcript
1
Accumulation-based Congestion ControlYong Xia, David Harrison†, Shivkumar Kalyanaraman,
Kishore Ramachandran, Arvind Venkatesan†
ECSE and CS† DepartmentsRensselaer Polytechnic Institute, Troy, NY 12180, USA
Abstract—This paper1 generalizes the TCP Vegas conges-tion avoidance mechanism and uses accumulation, bufferedpackets of a flow inside network routers, as a congestionmeasure based on which a family of congestion controlschemes can be derived. We call this model accumulation-based congestion control (ACC), which fits into the nonlin-ear optimization framework proposed by Kelly [20]. TheACC model serves as a reference for packet-switching net-work implementations. We show that TCP Vegas is one pos-sible scheme under this model. It is well known that Vegassuffers from round trip propagation delay estimation errorand reverse path congestion. We therefore design a newMonaco scheme that solves these problems by employingan out-of-band, receiver-based accumulation estimator, withthe support of two FIFO priority queues from (congested)routers. Comparisons between these two schemes demon-strate that Monaco achieves better performance than Vegasand does not suffer from the problems mentioned above. Weuse ns-2 simulations and Linux implementation experimentsto show that the static and dynamic performance of Monacomatches the theoretic results. One key issue regarding theACC model in general, i.e., the scalability of router bufferrequirements and a solution using a virtual queuing algo-rithm, are discussed and evaluated.
I. INTRODUCTION
Much research has been conducted to achieve stable,efficient and fair operation of packet-switching networks.TCP congestion control [18], a set of end-to-end mecha-nisms, has been widely acknowledged for its critical rolein maintaining stability of the Internet. Among them, TCPReno [2] infers network congestion by detecting packetloss that is assumably caused only by congestion; alter-natively, TCP Vegas [10] measures backlog, the numberof buffered packets inside routers along the path, to detectnetwork congestion and demonstrates better performancethan Reno. Unfortunately, Vegas has technical problemsinherent to its backlog estimator that prevent it from func-tioning properly. There has been a substantial amount of
1This work was supported by NSF under contracts ANI-9806660 andANI-9819112 and a grant from Intel Corp. Part of the paper has beenpresented at the IEEE International Conference on Communications inAnchorage, Alaska, USA on May 11-15, 2003.
work on this issue, such as [1] [26] [17] [8] [9] [25] [12]and references therein, which we review in Section II. Butnone of them provides a solution to estimate backlog un-biasedly in case of round trip propagation delay estimationerror or reverse path congestion.
In this paper, we offer a solution to this problem and de-velop a systematic model to generalize Vegas’ congestionavoidance mechanism. Formally, we define in a bit-by-bitfluid model the backlog (hereafter we call it accumulation)as a time-shifted, distributed sum of queue contributions ofa flow at a set of FIFO routers on its path. The central ideais to control flows’ rate by controlling their accumulationsin an end-to-end and distributed manner. We study a set ofclosed-loop congestion control schemes that are all basedupon the idea of keeping a target accumulation for eachflow individually.
The key concepts for this accumulation-based conges-tion control (ACC) model are developed in Section III.An ACC model has two components: congestion estima-tion and congestion response. The former defines a con-gestion measure (i.e., accumulation) and provides an im-plementable estimation of the measure; while the latterdefines an increase/decrease policy for the source throt-tle. A class of control algorithms, including the additive-increase/additive-decrease (AIAD) policy [11], Mo andWalrand’s proposal [27] and a proportional control, canbe used. Based on previous research [27] [20], we showthat the equilibria of all these algorithms achieve the sameproportional fairness in the appendix.
To instantiate the ACC model, choices can be made ineach of the ACC components to put together the entirescheme. We describe two example schemes in Section IV.We show that Vegas congestion avoidance attempts to esti-mate accumulation and thus fits into the ACC family. Butit often fails to provide an unbiased accumulation estimate.We therefore design a new scheme called Monaco that em-ulates the ACC fluid model in a better way. Particularly,Monaco solves the Vegas problem by employing an out-of-band, receiver-based accumulation estimator. We provideresolution to a number of concerns regarding accumula-tion estimation in Section IV-C. Section V demonstratesthe steady state and dynamic performance of Monaco us-
2
ing extensive ns-2 [28] simulations as well as Linux im-plementation experiments. Section III-C discusses a keyconcern regarding the ACC model in general, i.e., the scal-ability of buffer requirements resulting from accumulatingpackets in the congested router buffers for every flow. Sec-tion IV-D proposes a solution to this issue based on the vir-tual queuing algorithm in [23]. We conclude this paper inSection VI.
II. RELATED RESEARCH
The most closely related work starts from the TCP Ve-gas protocol, followed by a series of nonlinear optimiza-tion based models for network congestion control.
TCP Vegas [10] includes three new techniques: a modi-fied slow start, a more reactive retransmission mechanismresulting in less timeouts, and a new congestion avoidancewhich maintains a “right” amount of extra packets insidethe network. Its authors claim that Vegas achieves higherthroughput and less packet losses than Reno using simu-lations and Internet measurements, confirmed experimen-tally by Ahn et al. [1] and analytically by Mo et al. [26],who also point out Vegas’ drawbacks of estimating roundtrip propagation delay (RTPD) incorrectly in the presenceof rerouting and possible persistent congestion. Instead ofusing the minimum of all round trip time (RTT) samples asan estimation of RTPD, they suggest to use the minimumof only the most recent k RTT samples. As we discussed inSection IV-A, this estimation is still inflated because thereis always a steady state standing queue on the path.
Bonald compares Reno and Vegas by means of a fluidapproximation [8]. He finds that Vegas is more stable thanReno, resulting in a more efficient utilization of networkresources, and shares fairly the available bandwidth be-tween users with heterogeneous RTTs. But its fairnesscritically depends on accurate estimation of RTPD, con-firmed by the analysis of Boutremans et al. [9].
A different line of research of network congestion con-trol theoretic models is pioneered by Kelly’s optimizationframework [20], followed by Low et al. [24] and Srikantet al. [22], where they model congestion control as a non-linear optimization problem under which all users try tomaximize their own interest, subject to a set of capacityconstraints. Following Gibbens and Kelly’s work [14],Kunniyur and Srikant develop an Adaptive Virtual Queue(AVQ) algorithm [23], which we leverage in this paperto keep a low steady state queue in the congested router(see Section IV-D). Low, Peterson and Wang present anoptimization model for Vegas [25]. Then Jin, Wei andLow extend Vegas and design a FAST protocol for highbandwidth-delay-product networks [19]. Low et al. im-
d j
µij i,j+1
RJRj Rj+1R1
other
flows
flow ii µ i
ingress
edge
egress
edge(a) Network Model
)( ttai
)(1f
ii dtq)(
1J
jk
kij dtq
jd)(tqiJ
)(tai
f
id
t
time axis
… …
(b) Accumulation
Fig. 1. Network Fluid Model of Accumulation
prove Vegas performance using a Randomly ExponentialMarking (REM) buffer management algorithm [3]. Simi-lar to Vegas+REM, we use Monaco+AVQ in this paper asan alternative solution.
Mo and Walrand propose a fair end-to-end window-based scheme that includes a proportionally fair controlalgorithm [27]. However, this algorithm raises technicalchallenges in its practical implementation. Our Monacoaccumulation estimator can be thought as such an imple-mentation that requires two-separate-FIFO-queue supportfrom the congested routers.
III. ACC FLUID MODEL
In this section we describe the ACC model. We defineaccumulation under a bit-by-bit fluid model and use accu-mulation to measure and control network congestion. Inthe appendix we briefly prove that keeping a target accu-mulation inside routers for each flow is equivalent to a non-linear optimization that allocates network capacity propor-tionally fairly. We show that a set of control algorithmsexist for each flow to achieve its target accumulation.
A. Accumulation
Consider an ordered sequence of FIFO nodes {R1, . . . ,RJ} along the path of a unidirectional flow i in Figure 1(a).The flow comes into the network at the ingress node R1
and, after passing some intermediate nodes R2, . . . , RJ−1,goes out from the egress node RJ . At time t in any nodeRj (1 ≤ j ≤ J), flow i’s input rate is λij(t), output rate isµij(t). The propagation delay from node Rj to node Rj+1
is a constant value dj .2
2In practice R1/RJ can be mapped as a source/destination pair toform an end-to-end control loop or ingress/egress edge routers to forman edge-to-edge control loop. Here we focus on the ACC model itself.The reader is refer to [16] for discussion on architectural issues.
3
0
1e+06
2e+06
3e+06
4e+06
5e+06
6e+06
7e+06
8e+06
9e+06
0 1 2 3 4 5 6 7 8 9 10
Thro
ughp
ut (b
ps)
Time (secs)
Long FlowShort Flow
0
1e+06
2e+06
3e+06
4e+06
5e+06
6e+06
7e+06
8e+06
9e+06
0 1 2 3 4 5 6 7 8 9 10
Thro
ughp
ut (b
ps)
Time (secs)
Long FlowShort Flow
0
1e+06
2e+06
3e+06
4e+06
5e+06
6e+06
7e+06
8e+06
9e+06
0 1 2 3 4 5 6 7 8 9 10
Thro
ughp
ut (b
ps)
Time (secs)
Long FlowShort Flow
(a) ACC-AIAD (b) ACC-MW (c) ACC-P
Fig. 2. Different ACC algorithms achieve similar steady state bandwidth allocation, but the dynamic behavior differs significantly.
Define flow i’s accumulation as a time-shifted, dis-tributed sum of the queued bits in all nodes along its pathfrom the ingress node R1 to the egress node RJ , i.e.,
ai(t) =J
∑
j=1
qij(t −J−1∑
k=j
dk) (1)
where qij(t) is flow i’s queued bits in router j at time t, asillustrated as the solid slanted line in Figure 1(b). Notethe equation includes only those bits backlogged insidethe buffers of all nodes on the path, not those stored ontransmission links. (This definition provides a referenceto implement an unbiased accumulation estimator in Sec-tion IV-B.1.) We aim to control flow rates by controllingtheir accumulations, i.e., keeping a steady state accumula-tion inside the network for each individual flow.
B. Control Algorithms
In the ACC model we use accumulation to measurenetwork congestion as well as to probe available band-width. If accumulation is low, we increase the congestionwindow; otherwise, we decrease it to drain accumulation.More accurately, we try to maintain a constant target ac-cumulation a∗i for each flow i and a∗
i > 0. To achieve thisgoal we can choose from a set of control algorithms:3
i) ACC-AIAD additively increases and additively de-creases the congestion window value:
wi(t) = −κ
rtti· sgn(ai(t) − a∗i ) (3)
where wi(t), rtti, ai(t) and a∗i are respectively the conges-tion window size, round trip time, instantaneous accumu-
3All these algorithms fit into the following general form:
wi(t) = −η · g(t) · f(ai(t) − a∗
i ) (2)
where η > 0, g(t) > 0, f(·) is a function in the first and third quad-rants. It is nondecreasing and has a single root 0 (i.e., only f(0) = 0).
lation and target accumulation value of flow i, κ > 0 and
sgn(y) =
+1 if y > 00 if y = 0−1 if y < 0.
(4)
ii) ACC-MW is proposed by Mo and Walrand [27]:
wi(t) = −κ · rttpi
rtti·ai(t) − a∗i
wi(t)(5)
where rttpi is the round trip propagation delay of flow i.
iii) ACC-P is a proportional control policy that we usein this paper:
wi(t) = −κ
rtti· (ai(t) − a∗i ). (6)
Note all the above algorithms have the same single zeropoint ai(t) = a∗i . We present a set of algorithms herebecause they share a common steady state property of pro-portionally fair bandwidth allocation. We briefly state thisbelow and provide more details in the appendix.
C. Properties
For any congestion control, major theoretic concerns arestability, as well as fairness and steady state queue bound.Stability is to guarantee equilibrium operation of the algo-rithm. Fairness, either max-min [7] or proportional [20],determines the steady state bandwidth allocation amongcompeting flows. Steady state queue bound provides anupper limit on the router buffer requirement in equilibrium,which is important for real network deployment.
The stability of the general algorithm (2) is still an openquestion. So we turn to extensive simulations in Section Vto evaluate the stability of ACC-P.
Given that the equilibrium is achieved, we can provethat the equilibrium bandwidth allocation of ACC isweighted proportionally fair (See the appendix). Figure 2shows simulation results of a parking-lot topology of two9Mbps bottlenecks with one long flow and two short flows
4
(using the scheme developed in Section IV-B). It verifiesthat all the three algorithms do achieve similar steady statebandwidth allocation.4
Interestingly, different ACC control policies can havethe same fairness property. Thus to achieve a particularsteady state performance, we have the freedom to choosefrom a set of control policies that have different dynamiccharacteristics. In this sense, we remark that the ACCmodel manifests congestion control as a two-step issue ofsetting a target steady state allocation (fairness) and thendesigning a control algorithm (stability and dynamics) toachieve that allocation.
Even though we keep a finite accumulation inside thenetwork for every flow, the steady state queue at a bottle-neck scales up to the number of flows sharing that bot-tleneck. In practice, we need to provide enough bufferin the congested routers to avoid packet loss and makethe congestion control protocol robust to such loss, if un-avoidable (see Section IV-B). Another way to alleviate thisproblem is to control aggregate flow in a network edge-to-edge manner, instead of end-to-end for each micro-flowof source-destination pair, since the ACC model can bemapped onto end-to-end hosts or network edge-to-edge(though we focus on the model itself and don’t elaboratethe architecture issues in this paper). A possibly better so-lution to keep steady state queue length bounded is to usean active queue management (AQM) mechanism [13] suchas AVQ. We implement this option and discuss more de-tails in Section IV-D.
IV. ACC SCHEMES
Now we instantiate the ACC model into two schemesfor packet-switching networks. Firstly we show that TCPVegas tries to estimate accumulation and fits into the ACCmodel. But Vegas often fails to provide an unbiased ac-cumulation estimation. Then we design a new schemecalled Monaco which solves the estimation problems ofVegas. Monaco also improves the congestion response byutilizing the value of estimated accumulation, unlike Ve-gas’ AIAD policy that is possibly slow in reacting to asudden change in user demands or network capacity. Bycomparing Monaco and Vegas via analysis and simulationwe reach two observations: It is effective 1) to employa receiver-based mechanism and, 2) to measure forwardpath queuing delay, instead of round trip queuing delayas in Vegas, for an unbiased accumulation estimate. The
4More careful investigation of Figure 2 reveals that the equilibria ofthe three algorithms are not exactly the same. We believe this is due tothe burstiness in the discrete time simulation system that is not capturedby the continuous time fluid model.
scheme design is guided by the following goals:
Goal 1: Stability: The scheme should converge to anequilibrium in a reasonably dynamic environment withchanging demands or capacity;
Goal 2: Proportional Fairness: Given enough buffers,the scheme must achieve proportional fairness and oper-ate without packet loss at the steady state;
Goal 3: High Utilization: When a path is presented withsufficient demand, the scheme should converge around fullutilization of the path’s resources;
Goal 4: Avoidance of Persistent Loss: If the queueshould grow to the point of loss due to underprovisionedbuffers, the scheme must back off to avoid persistent loss.
A. Vegas
Vegas includes several modifications over Reno. How-ever, we focus only on its congestion avoidance mecha-nism, which fits well as an example ACC scheme.
The Vegas estimator for accumulation was originallycalled “backlog”, a term we use interchangeably in thispaper. For each flow, the Vegas estimator takes as inputan estimate of its round trip propagation delay, hereaftercalled rttp (or basertt in [10] [26]). Vegas then estimatesthe backlog as
aV = (expected rate − actual rate) × rttp
= (cwnd
rttp−
cwnd
rtt) × rttp (7)
which could be simplified as
aV =cwnd
rtt× rttq (8)
where cwnd/rtt is the average sending rate during thatRTT and rttq = rtt− rttp is the round trip queuing delay.If rttp is accurately available and there is no reverse pathqueuing delay, then according to Little’s Law, aV providesan unbiased accumulation estimate.
Vegas estimates rttp as the minimum RTT measured sofar. If bottleneck queues drain often, it is likely that eachcontrol loop will eventually obtain a sample that reflectsthe true propagation delay. The Vegas estimator is usedto adjust its congestion window size, cwnd, so that aV
approaches a target range of ε1 to ε2 packets. More ac-curately stated, the sender adjusts cwnd using a variantversion of the algorithm (3):
cwnd(n + 1) =
{
cwnd(n) + 1 if aV < ε1
cwnd(n) − 1 if aV > ε2(9)
5
where ε1 and ε2 are set to 1 and 3 packets, respectively.Vegas has several well-known problems:
• Rttp Estimation Errors: Suppose re-routing of a flowincreases its propagation delay. Vegas misinterprets suchan increase as less congestion and sends faster. Hence, thispolicy can lead to unbounded queue which introduces per-sistent loss and congestion [25], violating Goals 1 and 4.Mo et al. [26] suggest limiting the history on the rttp es-timate by using the minimum of the last k, instead of all,RTT samples. We refer to this variant as the “Vegas-k”scheme. Still, it cannot guarantee queue drain at interme-diate bottlenecks within k RTTs, shown in Section IV-C.
• Rttp with Standing Queues: When a flow arrives at abottleneck with a standing queue, it obtains an exagger-ated rttp estimate. The flow then adjusts its window sizeto incur an extra backlog between ε1 and ε2 packets in ad-dition to the standing queue. This leads to a bandwidthallocation away from the target proportional fairness, vio-lating Goal 2.
• Reverse Path Congestion: The Vegas estimator is af-fected by congestion in the reverse path. Reverse pathcongestion inflates the Vegas estimator leading to sharplyreduced utilization, not achieving Goal 3.
B. Monaco
Monaco emulates the accumulation defined by Equa-tion (1) and implements a receiver-based, out-of-bandmeasurement. It is immune to issues such as rttp sensi-tivities and reverse path congestion and robust to controland data packet losses. We describe the Monaco accumu-lation estimator and then its congestion response policy.
B.1 Monaco: Congestion Estimation Protocol
Let’s look at the definition of accumulation in Equa-tion (1). It is the sum of the queued bits of a flow at a se-quence of FIFO routers, including both ingress and egressnodes as well as intermediate routers. We aim to elimi-nate the computation at intermediate routers. Actually itis impossible for all nodes Rj (1 ≤ j ≤ J) to computesynchronously their queues qij(t −
∑J−1k=j dk) at different
times since no common clock is maintained.To estimate accumulation without explicit computation
at intermediate routers, Monaco generates a pair of back-to-back control packets once per RTT at the ingress nodeas shown in Figure 3. One control packet is sent out-of-band (OB) and the other in-band (IB). The OB control
packet skips queues in the intermediate routers by pass-ing through a separate dedicated high priority queue.5 As-suming the OB queues to be minimal as only other OBcontrol packets share them, such packets experience onlythe forward propagation delay df
i . The IB control packetgoes along with regular data packets and reaches the egressnode after experiencing the current queuing delay in thenetwork. The time interval between the OB and IB con-trol packets measured at the egress node is a sample ofthe current forward trip queuing time (fttq). Consideringa network with enough buffers where there is no packetloss, if flow rates at all routers do not change dramatically,then by Little’s Law, the number of data packet arrivals atthe egress node after the OB control packet, but before theIB control packet equals the accumulation. In Figure 3,the dashed lines cut by the forward direction OB controlpacket are those data packets, with each cut happeningin the router Rj at time t −
∑J−1k=j dk, ∀j ∈ {1, ..., J}.
Also observe in the figure that we can measure rtt at bothingress and egress nodes and rttp at the egress node.
Besides, we need to consider the effect of traffic bursti-ness. When we have a congestion window size cwnd, wealso compute a rate based on RTT estimation: rate =cwnd/rtt. At the ingress node we use this rate value tosmooth incoming traffic and thus alleviate the effect ofburstiness. At the egress node the accumulation is com-puted as the product of fttq and an exponentially weightedmoving average of the egress rate.
In practice, both data and control packets maybe lostbecause of inadequate router buffer size or too many com-peting flows. To enhance the robustness of Monaco esti-mator when data packets are lost, the IB control packet,identified by a control packet sequence number, carries abyte count of the number of data bytes sent during thatperiod. If the egress node receives fewer bytes than weretransmitted, then packet loss is detected. The forward OB
5An alternative implementation is to use IP Type of Service (TOS),i.e., assigning a low delay TOS to the high priority control packet ifTOS is supported in all (congested) routers.
6
control packet carries the same control packet sequencenumber as the associated IB control packet. Monaco sendscongestion feedback on the reverse OB control packet, inwhich there is one additional piece of information: conges-tion feedback, i.e., a flag denoting whether the congestionwindow cwnd should increase, decrease, or decrease-due-to-loss. The subsequent pair of forward control packets isgenerated after the arrival of the reverse OB control packetat the ingress node.
If either IB or OB control packet is lost, then the ingressnode times out and sends a new pair of control packets witha larger sequence number. The timer for control packet re-transmission is similar to that of TCP. These routine relia-bility enhancements are similar to those in the CongestionManager [4].
B.2 Monaco: Congestion Response Protocol
As already noted, we use accumulation to measure net-work congestion and to probe available bandwidth. Wekeep constant accumulation for every flow by increas-ing/decreasing its congestion window when the accumu-lation is lower/higher than the target value.
Since pure window-based control policy introduces un-desirable burstiness we use rate-paced window control tosmooth incoming traffic by employing at the ingress nodea leaky bucket shaper with a rate value of cwnd/rtt andburst parameter of one packet.
We provide below the Monaco’s proportional controlpolicy which is the discretized version of Equation (6):
cwnd(n + 1) = cwnd(n) − κ · (aM − a∗) (10)
where aM is the Monaco accumulation estimation, a∗, setto 3 packets, is a target accumulation in the path akin to ε1
and ε2 used by Vegas, κ is set to 0.5, and cwnd(n) is thecongestion window value at a control period n.
Monaco improves Vegas’ control policy by utilizing thevalue of estimated accumulation fedback by the reverseOB control packet, instead of taking it as binary infor-mation (i.e., “how congested”, instead of “congested ornot”). If the congestion feedback is decrease-due-to-loss,Monaco halves the congestion window as in TCP Reno.
C. Comparisons of Vegas and Monaco
Vegas and Monaco both aim to accurately estimateaccumulation, assuming different support from networkrouters. If rttp can be obtained precisely and there is noreverse path congestion then, by Little’s law, both of themgive unbiased accumulation estimation on average. But inpractice Vegas has severe problems in achieving this ob-jective; Monaco solves known estimation problems.
Monaco
Vegas
0
0.2
0.4
0.6
0.8
1
1.2
5 10 15 20 25 30 35 40 45
Bot
tlene
ck u
tiliz
atio
n
Reverse path bandwidth of the bottleneck (Mbps)
Fig. 4. Comparison between Vegas and Monaco under ReversePath Congestion
Vegas estimator operates at sender side. According toEquation (8) it actually calculates:
aV =cwnd
rtt× ( rtt − rttp ) (11)
=cwnd
rtt× ( tfq + tbq ) (12)
where tfq and tbq are forward and reverse direction queuingdelays, respectively. The above equations imply that Ve-gas may suffer from two problems: 1) By Equation (12),if there exists reverse direction queuing delay (because ofreverse direction flows), i.e., tb
q > 0, then Vegas overes-timates accumulation. This leads to underutilization andis hard to handle because the forward direction flows haveno control over those on reverse direction. To show thiseffect we use a simple dumb-bell topology with the bot-tleneck of 45Mbps forward direction bandwidth shared byseven forward direction flows and seven reverse flows. Wechange the bottleneck’s reverse direction bandwidth from5Mbps to 45Mbps. As shown in Figure 4, Vegas utilizationis only 10% ∼ 60%. 2) By Equation (11), if rttp is over-estimated, then Vegas underestimates accumulation. Thisleads to extra steady queue in bottlenecks or even persis-tent congestion. Results for a single bottleneck of 10Mbpsbandwidth and 12ms delay which is used by one flow em-ploying Vegas and Vegas-k are shown in Figures 5(a) and5(b), where rttp estimation error is introduced by a suddenpropagation delay change to 52ms at time 10s. Vegas oper-ates with very low utilization of less than 10% and Vegas-koperates with queue increase until loss occurs.
Due to the above problems, Vegas falls short of quali-fying as an effective ACC scheme, because we expect toachieve congestion control by maintaining constant accu-mulation for each flow at the steady state! In such a case,the sum of accumulations would lead to a non-zero steadystate queue which is not likely to drain, and hence dynamicrttp estimation would not possibly be unbiased with only
Fig. 5. Comparison between Vegas, Vegas-k and Monaco under rttp (or basertt) Estimation Error
in-band measurements. In summary, the rttp sensitivityissues of Vegas point to a fundamental problem of the in-band techniques for accumulation estimation.
Monaco solves both problems. Monaco estimator op-erates at receiver side and thus excludes the influence ofreverse path congestion. By measuring the time intervalbetween the OB and IB control packets, Monaco does notneed to explicitly estimate the forward direction propaga-tion delay. (Actually the forward path OB control packetimplicitly provides this value.) More specifically, sinceMonaco implements a rate-paced window control algo-rithm to smooth out incoming traffic, the time differencebetween the OB and IB control packet arrivals gives a sam-ple of the current forward direction queuing delay fttq.By Little’s law, the number of data packets arriving duringthis time period is the backlogged packets along the path.Using the OB control packet also makes Monaco adaptiveto re-routing since it is sent every RTT. As shown in Fig-ures 4 and 5(c), after a brief transient period of three sec-onds, Monaco operates at around 100% utilization with nopacket loss. So it’s immune to rttp estimation inaccuracyand reverse path congestion.
The above comparisons between Vegas, Vegas-k andMonaco suggest two observations on how to estimate ac-cumulation unbiasedly: 1) The key is to measure forwarddirection queuing delay (via the OB and IB control pack-ets in Monaco), instead of round trip queuing delay (asin Vegas); And consequently, 2) it’s better to measure ac-cumulation at the receiver side, otherwise it’s difficult toeliminate the effect of reverse path queuing delay, whichis hardly under forward direction congestion control.
D. Adaptive Virtual Delay Queuing
As we discussed in Section III-C there is a queue sizescalability problem for all the ACC schemes, includingboth Vegas and Monaco, since all of them keep non-zerosteady state accumulation inside the network for all flows.
According to the analysis in the appendix, the key to allACC schemes is to provide the queuing delay, or the La-grange multiplier from optimization perspective, which isa measure of network congestion. In a non-AQM droptailFIFO router, the Lagrange multiplier tql = ql/cl is pro-vided by a physical FIFO queuing process where cl is fixedand we have no freedom to control the physical queue ql.But similar to AVQ, we can provide the same value of theLagrange multiplier tql by running an AQM algorithm inthe bottleneck such that tql = q′l/c
′l if we adapt the virtual
capacity c′l appropriately (Also see discussions in [12]). Atthe same time the physical queue ql can be bounded.
So we leverage AVQ to emulate an adaptively changinglink capacity and compute a virtual queuing delay, which isdefined as the ratio of virtual queue length divided by vir-tual capacity and add it into the forward IB control packet.We call this mechanism Adaptive Virtual Delay (AVD) al-gorithm. A nice property of AVD is that it is incrementallydeployable since a mixed set of droptail and AVD routerscan work together. In such an environment the Monaco ac-cumulation estimation changes to aM = aDT + x · tAV D
where aDT is accumulation in those droptail bottlenecksmeasured between two control packets shown in Figure 3,x is the egress rate and tAV D is the sum of all virtual de-lays at those AVD bottlenecks.
8
S1
.
..
.
..
.
..
.
..
R0
1ms
1ms
1msS20
S11
All other links are 10Mbps.S=Source, D=Destination, R=Router.
Fig. 8. Monaco with the Same Buffer as the Above Case (55 packets) in an AVD Bottleneck
V. SIMULATIONS AND EXPERIMENTS
In the last section we have shown that Monaco out-performs Vegas. So we focus on evaluating the Monacoscheme using simulations and implementation experi-ments in this section. Our ns-2 simulations illustrate:
A) Dynamic behaviors such as convergence of through-put, instantaneous link utilization and queue length in Sec-tion V-A. We use a single bottleneck topology with het-erogeneous RTTs for tens of dynamic flows;
B) Steady state performance such as throughput fairnessin Section V-B. We use a linear topology of multiple con-gested links shared by a set of flows passing different num-ber of droptail and AVD bottlenecks.
We also implement Monaco in Linux kernel v2.2.18based on the Click router [21]. In Section V-C we useLinux implementation experiments to validate the ns-2simulation results. In all the simulations and experimentswe use the parameter settings in Table I. In brief, in combi-nation with Section IV-C, this section shows that Monaco
9
S1 D1
D2
D3
R0
S3
S2 R1
s c1 s c2
dc1 dcn4ms
12ms
20ms
S=Source, D=Destination, R=Router, N=2~9.All links are 100Mbps 4ms if unspecified.
RN
4
6
8
10
12
14
16
18
20
22
2 3 4 5 6 7 8 9
Thro
ughp
ut o
f lon
g flo
ws
(Mbp
s)
Num of bottlenecks
theory curvelong flow 1long flow 2long flow 3
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
2 3 4 5 6 7 8 9
Thro
ughp
ut C
.O.V
. of l
ong
flow
s (M
bps)
Num of bottlenecks
(a) Topology (b) Long Flow Average Throughput (c) Throughput Spread
Firstly we consider a single 30Mbps bottleneck with2ms propagation delay shared by three sets of flows us-ing Monaco, as shown in Figure 6(a). Set 1 has ten flowsstarting at 0s and stopping at 30s; Set 2 has five flows start-ing at 10s and stopping at 40s; Set 3 has five flows startingat 20s and stopping at 50s. Each source-destination pair isconnected to the bottleneck via a 10Mbps 1ms link. Theone-way propagation delays for the three sets of flows are4ms, 9ms and 14ms, respectively. We simulate for 50 sec-onds. We performed three simulations, the first one withenough buffer provided for a droptail bottleneck, the sec-ond one with underprovisioned buffer also for the droptailbottleneck, and the third one with an AVD bottleneck.
In the first simulation, the bottleneck router has enoughbuffer of 90 packets, as shown in Figure 6(d), where thereis no packet loss. We randomly pick one flow from each setand draw its individual throughput in Figure 6(e). We ob-serve that from 0s to 30s, the throughput is about 3Mbps,since only ten flows are active; When the five flows fromset 2 jump in at 10s, the throughput drops to 2Mbps, as
we have fifteen active flows. Similarly, when the finalfive flows from set 3 enter at 20s, the throughput changesto 1.5Mbps. Then at 30s, set 1 stops, the throughput in-creases to 3Mbps. At 40s, set 2 leaves, only the five flowsof set 3 are in the system with throughput of about 6Mbps.The congestion window dynamics is similar, as shown inFigure 6(c). Bottleneck queue length is depicted in Fig-ure 6(d) where incoming flows build up a steady queue andflows leave with queue decrease, on average 3 packets foreach flow. This matches the target accumulation specifiedas a control parameter in Table I. During the simulationbottleneck utilization always stays around 100%, excepttwo soon-recovered drops during abrupt demand changesat 30s and 40s as seen in Figure 6(b). From this simula-tion, we validate that Monaco demonstrates a stable be-havior under a dynamic and heterogeneous environmentand keeps steady queues inside bottleneck.
In the second simulation, the droptail bottleneck routerbuffer is underprovisioned, as illustrated in Figure 7(b),we can see that the queue length grows to the limit of thewhole buffer size of 55 packets, and there is a correspond-ing packet loss leading to halving of the congestion win-dow during 20s ∼ 30s. Consequently, the throughput ismore oscillating as seen in Figure 7(c), but the bottleneckis still fully utilized. From this simulation, we see thatwithout enough buffer, Monaco shows a degraded behav-ior under dynamically changing demands.
In the third simulation, the AVD bottleneck buffer isthe same as the second one. As illustrated in Figure 8(b),for most of time the bottleneck queue length is below 20packets. The throughput converges without oscillation as
10
Dst: C1, C2 Dst: C3, C4 Dst: C5, C6
100M100M
D1
……
1-100ms 1-100ms
R1 R2 R3
D2
1-100ms
1ms
100ms
R0
S1
S2
…
100M
…500 web
B users
Src: C5, C6
…500 web
A users
Src: C3, C4
…
Src: C1, C2
…
…
web B…
web A
…
Bandwidth for all unlabelled links are 1Gbps; Delay 1ms.
6e+06
8e+06
1e+07
1.2e+07
1.4e+07
1.6e+07
1.8e+07
2e+07
0 5 10 15 20 25 30 35 40 45 50
Thro
ughp
ut (b
ps)
Time (sec)
"Long_Flow_1""Long_Flow_2"
0
1e+07
2e+07
3e+07
4e+07
5e+07
6e+07
0 5 10 15 20 25 30 35 40 45 50
Thro
ughp
ut (b
ps)
Time (sec)
"Short_Flow_1""Short_Flow_4""Short_Flow_6"
(a) Topology (b) Long Flow Throughput (c) Short Flow Throughput
Fig. 10. Monaco with a Large Amount of Background Web Traffic under Multiple Mixed Droptail and AVD Bottlenecks
shown in Figure 8(c), comparable to result in the first sim-ulation. The bottleneck utilization is around 98%, whichis the target utilization value we configure in the AVDalgorithm. This simulation shows that the AVD mecha-nism is effective in controlling the queue size and thusmake Monaco more stable comparing to droptail bottle-neck without enough buffer provisioned.
B. Multiple Bottlenecks
Firstly we show the steady state performance of Monacowhen a flow traverses more than one bottleneck. We usea linear topology with multiple congested links depictedin Figure 9(a). We did a set of simulation experimentsby changing the number of bottlenecks N from 2 to 9.To show the compatibility of AVD in a droptail environ-ment, we randomly set some bottlenecks droptail and oth-ers AVD. There are three “long” flows passing all the bot-tlenecks and a set of “short” flows each using only one bot-tleneck. Every bottleneck link has 100Mbps capacity and4ms delay. The long flows have very different RTTs. Wesimulated for 50 seconds under only one condition withenough buffer provided for all the droptail routers. As al-ready shown in the last subsection, if droptail router bufferis underprovisioned, the Monaco performance degrades.
As illustrated in Figure 9(b), the steady state through-put curves of all long flows are located near the theoreticone of 100/(3 + N)Mbps. Each individual long flow getsroughly its fair share, for all cases of N = 2, 3, ..., 9 bot-tlenecks. The difference of throughput between the 3 longflows is measured by the Coefficient of Variance (C.O.V.)of their throughput, depicted in Figure 9(c), which is be-
tween 2% and 5% for all cases. For a particular simu-lation of five bottlenecks, we pick up two of them, onedroptail and one AVD, and draw their queue length in Fig-ures 9(d) and 9(e), respectively. Obviously the AVD bot-tleneck keeps a lower queue than the droptail. We showthe throughput convergence of two kinds of flows in Fig-ure 9(f), where after about 10s of transient period, the longflow’s throughput converges to 12Mbps (around its theo-retic fair share of 12.5Mbps), and the short flow’s to some65Mbps (around its theoretic fair share of 62.5Mbps). Thissimulation demonstrates that, with enough buffer provi-sioned, Monaco achieves a proportionally fair bandwidthallocation in a multiple bottleneck case, validating our the-oretic results in Section III.
Now we go further for a more realistic condition byadding web traffic into the former multiple bottleneck net-work. To simulate web traffic, we use Barford and Crov-ella’s HTTP model introduced in [5]. A three-bottlenecktopology is shown in Figure 10(a) where R1 is an AVDrouter and others are droptail. All bottleneck bandwidth is100Mbps, whereas access bandwidth is 1Gbps. The propa-gation delay for each link is also shown in the figure. Note“1-100ms” means that there are a number of links withpropagation delays evenly ranging from 1ms to 100ms.For example, for a two-link case, the propagation delaysare 1ms and 100ms, respectively. All unlabelled links have1ms delay. There are two long flows, three short-flow setseach of them includes two flows, and two web-traffic setswhich totally have one thousand web connections multi-plexed on fifty access links. Long flow 1 starts at 0s andstops at 50s, while long flow 2 starts at 20s and stops at
11
S1
R1
s c2
dc1
1Mbps 1Mbps
sc1
c2d
R 0 2R
S2 D2
D1
S=Source, D=Destination, R=Router.All other links are 10Mbps.
20ms 20ms
short flow 1 & 2
long flow 1
long flow 2
0
0.2
0.4
0.6
0.8
1
0 10 20 30 40 50 60 70 80
Thr
ough
put (
Mbp
s)
Time (sec)
0
0.2
0.4
0.6
0.8
1
0 10 20 30 40 50 60 70 80
Thr
ough
put (
Mbp
s)
Time (sec)
0
0.2
0.4
0.6
0.8
1
0 10 20 30 40 50 60 70 80
Thr
ough
put (
Mbp
s)
Time (sec)
0
0.2
0.4
0.6
0.8
1
0 10 20 30 40 50 60 70 80
Thr
ough
put (
Mbp
s)
Time (sec)
0
0.2
0.4
0.6
0.8
1
0 10 20 30 40 50 60 70 80
Thr
ough
put (
Mbp
s)
Time (sec)
(a) Topology (b) Flow Throughput
Fig. 11. Monaco Linux Kernel Implementation Results
40s. All short flows start randomly during 0∼3s. Webtraffic starts at 30s. We simulated for 50 seconds.
The throughput dynamics for the two long flows andthree randomly chosen short flows is shown in Fig-ures 10(b) and 10(c) which demonstrate that both long andshort flows’ throughput rates converge respectively ontotheir theoretic values, i.e., 1 (or 3)×100Mbps
6+number of active long flows
for the long (or short) flows. For example, the long flow1 gets 14Mbps during 0∼20s, 12.4Mbps during 20s∼30s,drops a little when web traffic comes in and then goes up toabout 13Mbps after the long flow 2 leaves at 40s; whereasthe short flow 1 gets 43Mbps during 3∼20s, 38Mbps dur-ing 20∼30s, and then a little more than those because theweb traffic at the bottlenecks R1 → R2 and R2 → R3 lim-its the throughput of the long flows. The queue length ofthe AVD bottleneck R1 is shown in Figure 10(d), while thedroptail bottleneck R2’s is depicted in Figure 10(e). Ob-viously AVD keeps a low and constant queue, while thedroptail queue is sensitive to the number of flows. Evenwhen web traffic jumps in, which makes the AVD queuemore oscillating, it is still much lower than the droptailqueue. In summary, this simulation shows that Monacoworks in an environment of multiple bottlenecks with dy-namic demands and bursty background traffic.
C. Implementation Results
We did a set of experiments using our Monaco Linuximplementation to validate the stability and fairness resultsfrom ns-2 simulations in the last two subsections. Herewe show one result for a two-bottleneck topology with dy-namic demands. For more details, the reader is referred to[30]. We have 2 droptail bottlenecks each of 1Mbps band-width and 20ms delay as drawn in Figure 11(a). Duringthe 80s of experiment, we have 2 short flows always active,one long flow coming in at 20s and going out at 60s, andanother long flow active from 40s to 80s. After a brief tran-sient period, each flow stabilizes at its proportionally fairshare, illustrated by Figure 11(b). For instance, the firstlong flows’ throughput starts with 0.33Mbps (its propor-tionally fair share) at 20s and changes to some 0.25Mbps
at 40s when the second long flow shows up. At the sametime, the short flows get 0.5Mbps, dropping from their for-mer throughput of about 0.65Mbps. After 60s, the secondlong flow gets about its fair share of 0.33Mbps.
Comparing with the simulation throughput results inSection V-A, implementation results in Figure 11(b) aremore oscillating. This comes mainly from limited timergranularity in Linux kernel which makes traffic less regu-lated (more bursty) than in ns-2 simulations.
VI. SUMMARY
In this paper we generalize TCP Vegas and develop amodel using accumulation, buffered packets of a flow in-side the network routers, as a measure to detect and controlnetwork congestion. Applying Mo and Walrand’s queuinganalysis and Kelly’s nonlinear optimization framework onthe model (in the appendix), we show that ACC allocatesnetwork bandwidth proportionally fairly at the equilibrium– which is its steady state feature. A set of control al-gorithms can drive the network to the same equilibrium– this is related to its dynamic characteristics. A familyof schemes, including Vegas, could be derived based onACC. Using the model as a reference, we design a newMonaco scheme which, with two priority FIFO queuesprovided by (congested) network routers, solves the well-known problems of Vegas. In particular, using analysisand simulations, we show that the receiver-based, out-of-band estimator is able to produce an unbiased accumula-tion measurement.
Using extensive ns-2 simulations, we evaluate the dy-namic and steady state performance of Monaco under dif-ferent topologies and conditions. The scheme demon-strates its effectiveness in keeping network stable, fair,and efficiently utilized, given enough buffers in the bot-tlenecks. With underprovisioned buffer, Monaco’s perfor-mance degrades. This buffer scalability problem can besolved by running the AVQ algorithm inside the bottle-necks, which works compatibly with the non-AQM drop-tail routers. We implement Monaco in Linux kernel based
12
on the Click router and validate most of the simulation re-sults on an internal testbed and the Utah Emulab [29].
In summary, the main contributions of this paper are:
• a model of accumulation-based congestion control basedon which a family of schemes could be derived;
• a Monaco scheme implemented as a packet-switchingnetwork protocol which estimates accumulation unbias-edly and utilizes this value in a non-binary manner to con-trol congestion;
• a comparison between Vegas and Monaco showing thatMonaco’s receiver-based, out-of-band accumulation mea-surement solves Vegas’ well-known estimation problems;
• an incrementally deployable virtual delay queuing algo-rithm based on AVQ as a solution to the problem of un-bounded router buffer size requirement.
One may ask that if the two-queue support from all thebottlenecks, even its complexity is very low, is unrealistic.Firstly, this requirement is largely eliminated at an AVDbottleneck which provides virtual delay, instead of usingphysical queuing delay, as its congestion measure. Sec-ondly, for a non-AQM droptail bottleneck, as already ex-plored in related research and this paper, in-band measure-ment techniques suffer from inherently hard accumulationestimation problem. So there is a fundamental tradeoff be-tween an ACC scheme’s performance and its requirement.
By keeping different accumulation for different flows,it’s possible to provide differentiated services, such as min-imum bandwidth assurance and weighted rate differentia-tion. These issues are explored in our related work [16].
VII. ACKNOWLEDGMENT
The authors thank Prof. Steven Low of Caltech for dis-cussions on REM. Thanks to their colleagues at RPI: JoshHort, Sthanu Ramakrishnan and Rahul Sachdev for theirrelated work, Prof. Murat Arcak and Xinzhe Fan for help-ful discussions, and Prof. Biplab Sikdar and Kerry Woodfor proofreading a draft version of this paper. They aregrateful to the anonymous reviewers and the editor fortheir insightful comments.
REFERENCES
[1] J. Ahn, P. Danzig, Z. Liu and L. Yan. Evaluation of TCP Vegas:Emulation and Experiment. Proc. SIGCOMM’95, August 1995.
[2] M. Allman, V. Paxson and W. Stevens. TCP Congestion Control.IETF RFC 2581, April 1999.
[3] S. Athuraliya, V. Li, S. Low and Q. Yin. REM: Active QueueManagement. IEEE Network, 15(3):48-53, May 2001.
[4] H. Balakrishnan, H. Rahul and S. Seshan. An Integrated Con-gestion Management Architecture for Internet Hosts. Proc. SIG-COMM’99, September 1999.
[5] P. Barford and M. Crovella. A Performance Evaluation of HyperText Transfer Protocols. Proc. SIGMETRICS’99, March 1999.
[6] M. Bazaraa, H. Sherali and C. Shetty. Nonlinear Programming:Theory and Algorithms. 2nd Ed., John Wiley & Sons, 1993.
[7] D. Bertsekas and R. Gallager. Data Networks. 2nd Ed., Simon &Schuster, December 1991.
[8] T. Bonald. Comparison of TCP Reno and TCP Vegas via FluidApproximation. INRIA Tech Report, November 1998.
[9] C. Boutremans and J. Le BoudecT. A Note on the Fairness ofTCP Vegas. Proc. of International Zurich Seminar on BroadbandCommunications, February 2000.
[10] L. Brakmo and L. Peterson. TCP Vegas: End to End CongestionAvoidance on a Global Internet. IEEE Journal on Selected Areasin Communications, 13(8):1465-1480, October 1995.
[11] D. Chiu and R. Jain. Analysis of the Increase/Decrease Algo-rithms for Congestion Avoidance in Computer Networks. Journalof Computer Networks and ISDN, 17(1):1-14, June 1989.
[12] D. Choe and S. Low. Stabilized Vegas. To Appear in Proc. INFO-COM’03, April 2003.
[13] S. Floyd and V. Jacobson. Random Early Detection Gatewaysfor Congestion Avoidance. IEEE/ACM Trans. on Networking,1(4):397-413, August 1993.
[14] R. Gibbens and F. Kelly, Resource Pricing and the Evolution ofCongestion Control. Automatica, 35:1969-1985, 1999.
[15] R. Guerin, S. Kamat, V. Peris and R. Rajan. Scalable QoS Provi-sion Through Buffer Management. Proc. SIGCOMM’98, Septem-ber 1998.
[16] D. Harrison, Y. Xia, S. Kalyanaraman and A. Venkatesan. AClosed-loop Scheme for Expected Minimum Rate and WeightedRate Services. Http://www.rpi.edu/˜xiay/pub/acc qos.ps.gz, Sub-mitted, 2002.
[17] U. Hengartner, J. Bolliger and T. Gross. TCP Vegas Revisited.Proc. INFOCOM’00, March 2000.
[18] V. Jacobson. Congestion Avoidance and Control. Proc. SIG-COMM’88, August 1988.
[19] C. Jin, D. Wei and S. Low. HighSpeed TCP for Large CongestionWindows. IETF draft-jwl-tcp-fast-01.txt, June 2003.
[20] F. Kelly, A. Maulloo and D. Tan. Rate Control in Communica-tion Networks: Shadow Prices, Proportional Fairness and Sta-bility. Journal of the Operational Research Society, Vol.49, pp.237-252, 1998.
[21] E. Kohler, R. Morris, B. Chen, J. Jannotti, and F. Kaashoek.The Click Modular Router. ACM Trans. on Computer Systems18(3):263-297, August 2000.
[22] S. Kunniyur and R. Srikant. End-To-End Congestion Control:Utility Functions, Random Losses and ECN Marks. Proc. INFO-COM’00, March 2000.
[23] S. Kunniyur and R. Srikant. Analysis and Design of an AdaptiveVirtual Queue (AVQ) Algorithm for Active Queue Management.Proc. SIGCOMM’01, August 2001.
[24] S. Low and D. Lapsley. Optimization Flow Control, I: BasicAlgorithm and Convergence. IEEE/ACM Trans. on Networking,7(6):861-875, December 1999.
[25] S. Low, L. Peterson and L. Wang. Understanding TCP Vegas: ADuality Model. Proc. SIGMETRICS’01, June 2001.
[26] J. Mo, R. La, V. Anantharam and J. Walrand. Analysis and Com-parison of TCP Reno and Vegas. Proc. INFOCOM’99, March1999.
[27] J. Mo and J. Walrand. Fair End-to-End Window-based Congestion
13
Control. IEEE/ACM Trans. on Networking, 8(5):556-567, Octo-ber 2000.
[28] Network Simulator ns-2. Http://www.isi.edu/nsnam/ns/.[29] Utah Emulab. Http://www.emulab.net/.[30] A. Venkatesan. An Implementation of Accumulation-based Con-
gestion Control Schemes. RPI M.S. Thesis, August, 2002.
APPENDIX
We apply Mo and Walrand’s queuing analysis [27] andKelly’s nonlinear optimization framework [20] to demon-strate the equilibrium characteristics of the ACC modeldescribed in Section III. It turns out that, given an ap-propriate control algorithm, ACC steers the network to anequilibrium of proportional fairness.
Network congestion control can be formalized as a re-source allocation problem. Consider a network of a setL = {1, . . . , |L|} of links, shared by a set I = {1, . . . , |I|}of flows. Each link l ∈ L has capacity cl. Flow i ∈ Ipasses a route Li consisting of a subset of links, i.e.,Li = {l ∈ L | i traverses l}. A link l is shared by a subsetIl of flows where Il = {i ∈ I | i traverses l}. Obviouslyl ∈ Li if and only if i ∈ Il.
Let’s firstly consider from queuing perspective [27]. Af-ter the system approaches a steady state (so we can neglectthe time variable t in all the previous equations), at anylink l the queue length ql (=
∑
i∈Ilqil), or equivalently the
queuing delay tql (= ql/cl), could be non-zero only if thecapacity cl is fully utilized by the sharing flows of the ag-gregate rate
∑
i∈Ilxi, where xi is the sending rate of flow i.
This suggests either ql = 0 (i.e., tql = 0 which means thelink is not congested) or
∑
i∈Ilxi = cl (which means the
link is congested). We use window-based congestion con-trol, in which a window wi bits of flow i could be storedeither in node buffers as accumulation ai (=
∑
l∈Liqil) or
on transmission links as xi · rttp i, where rttp i is flow i’sround trip propagation delay. Note wi = xi · rtti, wherertti is the round trip time observed by flow i. We summa-rize the above analysis to get the following result:
Proposition 1: If we use accumulation ai as a steeringparameter to control flow i’s congestion window size wi,then at the steady state we have, ∀i ∈ I,∀l ∈ L :(a) wi = ai + xi · rttp i, i.e., ai = xi(rtti − rttp i) =xi ·
∑
l∈Litql;
(b) tql · (cl −∑
i∈Ilxi) = 0;
(c)∑
i∈Ilxi ≤ cl;
(d) tql ≥ 0;(e) xi > 0.
Alternatively, network resource allocation can also bemodelled as an optimization problem [20] [24] [22], wherethe system tries to maximize the sum of all flows’ utility
functions∑
i∈I Ui(xi), in which flow i’s concave utilityfunction Ui(xi) is a measure of its welfare when it sendsat a rate of xi > 0, subject to a set of capacity con-straints
∑
i∈Ilxi ≤ cl at all the links. Using the Lagrange
multiplier method [6], we construct a Lagrange functionL(x, p) =
∑
i∈I Ui(xi) +∑
l∈L pl · (cl −∑
i∈Ilxi). If
utility functions are defined as Ui(xi) = si lnxi, wheresi > 0 is a weight, then because of the strict concavityof the objective function constrained by a convex set, theKarush-Kuhn-Tucker condition can be applied to obtain:
Proposition 2: The nonlinear programming problem
maximize∑
i∈I
si lnxi (13)
subject to∑
i∈Il
xi ≤ cl,∀l ∈ L
xi > 0,∀i ∈ I
has a unique global maximum. The sufficient and neces-sary condition for the maximum is, ∀i ∈ I,∀l ∈ L :(a) ∂L(x, p)/∂xi = 0, i.e., si = xi ·
∑
l∈Lipl;
(b) pl · (cl −∑
i∈Ilxi) = 0;
(c)∑
i∈Ilxi ≤ cl;
(d) pl ≥ 0;(e) xi > 0.
Now let’s compare the above two results. If replacingsi with ai, pl with tql, we find that Proposition 2 is turnedinto Proposition 1, and vice versa. This observation indi-cates that, by using accumulation as a steering parameterto control flow rate, the network is actually doing a non-linear optimization in which flow i’s utility function is
Ui(xi) = ai lnxi. (14)
It turns out that accumulation ai is an instance of theweight si, which could be used to provide a weighted pro-portionally fair congestion control. Besides, the Lagrangemultiplier pl is a measure of congestion, or price exploredin [24], at link l. In particular, the queuing delay tql is aninstance of such price. The more severe the congestion atlink l, the higher the price pl, the larger the queuing delaytql. If there is no congestion at that link, then there is noqueuing delay at all, i.e., tql = 0, the price pl is also 0.
Given the above utility function of ACC, it is straight-forward to show that its equilibrium bandwidth allocationis weighted proportionally fair where the accumulation ai
