1

Can coarse circuit switching work & What to do when it doesn't?

Jerry ChouAdvisor: Bill Lin

University of California, San Diego

CNS Review, Jan. 14, 2009

2

Outline

• Motivation

• Overview of new optical networking paradigm

• How to provision optical circuits?

• What to do when provision circuits not enough?

• Conclusions

3

Internet Traffic Ever Increasing

4

Current Packet Routing Scenario

• Packets electronically routed hop-by-hop– IP routers interconnected over switched optical backbone– OEO conversion and queuing delays at each hop

OXC

OXC

OXC

OXC

OXC

5

Optical Circuit Switching

• If optical circuit switching would work, then no intermediate per-hop queuing delays and OEO conversions = much faster

OXC

OXC

OXC

OXC

OXC

6

Optical Switching Options

• Extremely difficult to implement packet buffers and logic in optics

• No viable dynamically reconfigurable active optical switches at this time scale

PacketSwitching

10 ns

7

Optical Switching Options

• New signaling protocol and electronic control plane required to implement dynamic reservations

• Although active optical switches available at this time scale, coordination of such frequent network-wide reconfigurations not easy

PacketSwitching

10 ns

OpticalBurst

Switching

1 ms

8

Optical Switching Options

• Can we reasonably predict the traffic so that we can provision optical circuits to carry them?

• Can we provide a “fall-back” mechanism when circuit capacity is enough?

PacketSwitching

10 ns

OpticalBurst

Switching

1 ms

Quasi-StaticOpticalCircuits

1 hr

Over 3 Million X

9

Outline

• Motivation

• Overview of new optical networking paradigm

• How to provision optical circuits?

• What to do when provision circuits not enough?

• Conclusions

10

Observation

• Aggregate traffic at the core is relatively smooth and variations are predictable

Source: Roughan’03 on a Tier-1 US Backbone

11

Case Study

• On high-performance public backbone networks– Abilene (US):11 nodes, 23 links– GEANT (Europe): 23 nodes, 74 links– Public traffic matrices are available

• Optical circuits only change on hourly basis

• Use historical traffic to “predict” how much traffic will occur in the future– Abilene: 03/01/04-04/21/04, GEANT: 01/01/05–04/10/05

• Provision circuits to maximize likelihood that circuits have enough capacity

• Simulated actual traffic (over a week)– Abilene: 04/22/04-04/26/04, GEANT: 04/11/05–04/15/05

12

Circuits

• Setup circuits possibly across multiple paths in physical layer

Seattle

Sunnyvale

Indianapolis

Denver

Los Angeles Kansas City

ChicagoNew York

Washington

Atlanta

Houston

13

Circuits

• Logically one (optical) circuit for each OD-pair (origin-destination pair)

Seattle

New York

14

Abilene Network• Drop rates is the percentage of offering traffic exceeding its

circuit capacity• To consider a highly utilized network, traffic is scaled, such that at

least one link is saturated under OSPF• Worst-case 6.41%, 0.33% on average, mostly at or near 0%

Circuit switching works “most of the time” if carefully provisionedCircuit switching works “most of the time” if carefully provisioned

15

New Paradigm• Provision optical circuits that maximize the probability

of sufficient capacity to carry traffic

• Use optical circuit switching by default

• When actual traffic exceeds circuit capacities, route (electronically) over other “pre-configured circuits” with spare capacity

OXCOptical transit traffic

Traffic arriving tointermediate node

Smaller (simpler) routers

16

Analogy

• Direct “non-stop” flights (optical circuits) by default• If overbooked, re-route (electronically) excess demand

through alternative multi-hop flights

Seattle NY

Houston

To:NY

To:HS

To:NY

17

Abilene Network

• No packet drops with re-routing (adaptive load-balancing method to be discussed)

18

Advantages of New Paradigm

• Minimize queuing delay and latency for packets

• Reduce workload on electronic routers

• Optical circuits change infrequently, and mechanisms exist to provision circuits

• Key idea is to re-route electronically excess traffic rather than “on-the-fly” dynamic optical circuit reconfigurations

• Avoid new signaling protocol and frequent coordination of network-wide reconfigurations

19

Outline

• Motivation

• Overview of new optical networking paradigm

• How to provision optical circuits?

• What to do when provision circuits not enough?

• Conclusions

20

Basic Idea

• Use historical traffic data sets to decide on bandwidth allocation– Major ISPs have data collection infrastructure already

21

Ideally, Traffic is Stable

• Abilene– 11 nodes connected by 10Gb/s links

Seattle

Sunnyvale

Indianapolis

Denver

Los Angeles Kansas City

Chicago New York

Washington

Atlanta

Houston

Seattle/NY:Always 5Gb/sAllocate: 5Gb/s

Sunnyvale/Houston:Always 5Gb/sAllocate: 5Gb/s

Both flows can be carried by provisioned circuits

22

But, Flows Fluctuate Differently

• Abilene– 11 nodes connected by 10Gb/s links

Seattle

Sunnyvale

Indianapolis

Denver

Los Angeles Kansas City

Chicago New York

Washington

Atlanta

Houston

Seattle/NY:High traffic meanLow traffic variance

Sunnyvale/Houston:Low traffic meanHigh traffic variance

Give more bandwidth to flows with “high mean” or “high variance”?

23

Circuit Provisioning Approach

• Use Cumulative Distribution Function (CDF) as “utility function” (predictor of “acceptance probability”)

• Acceptance probability– The probability of a provisioned circuit with enough capacity

to carry its offering traffic

24

Example

• Abilene– 11 nodes connected by 10Gb/s links

Seattle

Sunnyvale

Indianapolis

Denver

Los Angeles Kansas City

Chicago New York

Washington

Atlanta

Houston

Seattle/NY:90% time ≤ 6Gb/s50% time ≤ 4Gb/sAllocate: 6Gb/s

Sunnyvale/Houston:90% time ≤ 6Gb/s80% time ≤ 4Gb/sAllocate: 4Gb/s

Seattle/NY has 90% acceptance probability

Sunnyvale/Houston has 80% acceptance probability

25

Circuit Provisioning Approach

• Formulate bandwidth allocation (circuit provisioning) as multi-path utility max-min fair allocation problem

– Utility functions represent traffic statistics (generally utility functions can be non-linear)

– Max-min fairness reach balance between throughput and fairness

– Multi-path circuits provide more freedom and better performance

We provide the first solution to the multi-path utility max-min fair

allocation

We provide the first solution to the multi-path utility max-min fair

allocation

26

Multi-path Utility Max-min Algorithm• Allocation based on “water-filling algorithm” and

maximum concurrent flow

• Steps:1. Identify maximum common utility increment 2. Solve maximum concurrent flow problem to find multi-

path routing3. Identify saturated flow

Max utility

Fill-up by with a routing

Saturated flow

27

Multi-Path vs. Single-Path

• Significantly lower drop probability– Mean drop rate: 3.56% vs. 20.34%– Max drop rate: 18.25 vs. 34.72%

28

Outline

• Motivation

• Overview of new optical networking paradigm

• How to provision optical circuits?

• What to do when provision circuits not enough?

• Conclusions

29

r(C) = 20

• Localized approach:– load-balance on outbound circuits, weighted by

spare capacity

r(B) = 30

r(D) = 25

B

C

DA

1. r(B) < B[A, B] ?YES

NO

2. k = random (wk)

Optical Circuit

3535

35

Problem1: greedy solution based only one-hop info.Problem2: oscillation of weight changes can happen

Problem1: greedy solution based only one-hop info.Problem2: oscillation of weight changes can happen

Adaptive Load-Balanced Routing

30

Adaptive Load-balance Re-routing

• Distributed approach: Step1: Compute path cost by Distance-Vector-like protocol Step2: Update weights to reach Wardrop Equilibrium state

– Every interval only shift weight by a small fraction δ– Achieve fast converge and prevent oscillation– Based on selfish routing no coordination among nodes

s t1

1

4

32

5

1

12 1

Current weights: w1, w2

δ = f (C1, C1, w1, w2)w1 = w1 + δ, w2 = w2

- δ

path1 cost(C1): (1+4)=5path2 cost(C2): (1+8)=9

31

Abilene Network

• 90 percentile drop rate comparison– OSPF has 0% drop at scale factor of 1

32

Abilene Network

• 90 percentile drop rate comparison– Cisco’s “ecmp” load-balances across equal cost shortest

paths and achieve lower drop rate

33

Abilene Network

• 90 percentile drop rate comparison– Without rerouting, we suffer small drop rates even at the

scale factor of 1– But show lower drop rates at larger scale factors b.c of

greater path diversity and better load-balance

34

Abilene Network

• 90 percentile drop rate comparison– Achieve lowest drop rates among all– With rerouting, we don’t have drop until at a factor of

1.75.

35

Abilene Network

• Circuit provisioning achieve lower drop rates under high traffic load b.c of load-balanced routing path

• Rerouting effectively reduce drop rates under low traffic load by utilizing residual network capacity

36

Outline

• Motivation

• Overview of new optical networking paradigm

• How to provision optical circuits?

• What to do when provision circuits not enough?

• Conclusions

37

Conclusion• A new paradigm of optical circuit switching by default,

packet routing when necessary

• Formulate circuit provisioning as an utility max-min fair allocation problem and provide the first solution under multiple paths scenario

• Apply a adaptive load-balance protocol on re-routing

• Conduct empirical study on two backbone networks, Abilene and GEANT

• Show more than 95% of traffic can be carried by the network with carefully static circuit provisioning & all traffic can be routed after re-routing

38

Publication

• Jerry Chou, Bill Lin, "Coarse Optical Circuit Switching by Default, Rerouting over Circuits for Adaptation,“ Journal of Optical Networking, vol. 8, no. 1, pp. 33-50 (2009).

39

Thank You

40

Backup Slides

41

Work-In-Progress

• Capacity planning

• Fault-tolerance

• Better adaptive routing algorithms

• Joint circuit-provisioning and routability optimization

42

Motivation

• Traffic growing nearly twice rate of Moore’s Law– Difficult for electronic packet routers to keep up

• On the other hand, optical switching provides abundance of transmission capacity (e.g. WDM)– Rate of increase in optical transport capacity keeping pace

with traffic growth (with 100 Gbps per wavelength in next generation), well above Moore’s Law

– Rate of decrease in cost per unit of optical transport capacity well below Moore’s Law

43

Networks• Traffic used for prediction (over months)

– Abilene: 03/01/04 - 04/21/04, GEANT: 01/01/05 – 04/10/05

• Optical circuits only change on hourly basis (method to be discussed)

• Simulated actual traffic (over a week)– Abilene: 04/22/04 - 04/26/04, GEANT: 04/11/05 – 04/15/05

• To consider a highly utilized network, we scaled traffic by a factor, such that at least one link is saturated under OSPF.– Abilene: 4, GEANT: 2

44

Questions

• How to decide on circuit provisioning to maximize probability that the circuits provide sufficient capacity to carry traffic?– Formulated as a multi-path utility max-min fair bandwidth

allocation problem

• What to do when circuit capacity is not enough?– Adaptive load-balancing over circuits that have spare

capacity

45

Multi-Path Utility Max-Min Algorithm

• Based on water-filling algorithm and maximum concurrent flow (MCF) solver

1. Determine bandwidth allocation that achieves the maximum common utility for all flows

2. Determine path distribution by MCF routing

3. Identify saturated flows and fix their utility

Max utility

Fill-up by with a routing

Saturated flow

46

Binary Search• Find maximum utility by binary search over [0, 1]

– Determine flow traffic by utility functions– Find feasible route by querying a MCF solver

• If <1, decrease utility, otherwise increase utility

20

2010 30 40 50

4060

80

100

20

2010 30 40 50

4060

80

100

20

2010 30 40 50

4060

80

100

20

2010 30 40 50

4060

80

100

BW BW BW BW

Utility(%

)

Utility(%

)

Utility(%

)

Utility(%

)

C = 100Max utility Traffic

1 (50,50,50,50) 0.5

.0.6 (10,40,10,40) 1

0.5 (10,30,10,40) 1.25

47

Piece-Wise Linear Search

• Approximate utility functions as piecewise linear functions

• Replace binary search by searching through each piecewise linear segment– Query MCF by the inverse of slope as traffic– is proportional to maximum utility

Seg I

Seg II

Seg III

Seg IV

20

2010 30 40 50

406080

100

BW

Utility(%

)

20U[1

] - U[0

]

BW[1]-BW[0]10 20 30 40

0

105.0

10u

48

Identifying Saturated Flows

• By residual capacity is not enough– Miss-identified saturated flow in earlier iteration would

produce smaller bandwidth allocation

A

B

C

D

E

F

Let link capacity = 10

Bandwidth requirement: AE = 5, AF = 5

If select path ACDF, AE is saturated

If select path ABDF, AE is not saturated

49

Identifying Saturated Flows

• A flow is saturated if its utility cannot be increased by any feasible routing

• To guarantee optimality, flows have to be re-routed

50

Multi-Path vs. Single-Path

• Significantly higher utility– Minimum utility 92.90% vs. 74.74%

51

Avoiding Cycles

• Problem: packets may go in circles– Never reach destination– Waste circuit capacity

• One solution is to limit “time-to-live” (TTL)

• Alternatively, ensure “loop-free” routingby routing table construction

52

Loop-Free Routing Tables

• For OD-pair, solve maxflow to derive largest “ayclic” graph on “circuit”

• Build routing tables using both “source” and “destination” prefixes

s

t

53

Current Contributions

• New paradigm of optical circuit switching by default, packet routing when necessary

• First solution to the multi-path utility max-min fair bandwidth allocation problem

• Though not presented, utility max-min fair solver has been applied to a Denial-of-Service network security problem

54

r(C) = 15r(C) = 20r(C) = 25

Localized Adaptive Re-routing

• Basic idea: load-balance on outbound circuits, weighted by spare capacity

r(B) = 30

r(D) = 25

B

C

DA

1. r(B) < B[A, B] ?YES

NO

2. k = random (wk)

Optical Circuit

3535

35

r(B) = 35

Problem1: greedy solution based only one-hop info.Problem2: oscillation of weights could occur

Problem1: greedy solution based only one-hop info.Problem2: oscillation of weights could occur

55

Distributed Adaptive Re-routing

• Basic idea: 1. Collect path info. by a Distance-Vector-like protocol 2. Load-balance outgoing weights based on path cost

s t1

1

4

32

4

1

1

2 1

56

Distributed Adaptive Re-routing

Step1: Compute path cost– Every router measure downstream link cost– Exchange info. by a Distance-Vector-like protocol

s t

cost: 1

1

1

4

32

4

1

1

2 1

cost: 1

57

Distributed Adaptive Re-routing

Step1: Compute path cost– Every router measure downstream link cost– Exchange info. by a Distance-Vector-like protocol

s t

cost: 1+1=2

1

1

4

32

4

1

1

2 1

cost: 4+1=5

cost: 3+1=4

58

Distributed Adaptive Re-routing

Step1: Compute path cost– Every router measure downstream link cost– Exchange info. by a Distance-Vector-like protocol

s t

cost: 2+2=4

1

1

4

32

5

1

1

2 1

If weights are equalCost: (2+4)*0.5 +(5+5)*0.5 = 8