11

Online Balancing of Online Balancing of Range-Partitioned Data Range-Partitioned Data

with Applications to P2P Systemswith Applications to P2P Systems

Prasanna GanesanPrasanna Ganesan

Mayank BawaMayank Bawa

Hector Garcia-MolinaHector Garcia-Molina

Stanford UniversityStanford University

22

MotivationMotivation

Parallel databases use range partitioningParallel databases use range partitioning

Advantages: Inter-query parallelismAdvantages: Inter-query parallelism– Data Locality Data Locality Low-cost range queries Low-cost range queries

High thru’putHigh thru’put

0 10035 60 8020Key

Range

33

The ProblemThe Problem

How to achieve load balance?How to achieve load balance?– Partition boundaries have to change over Partition boundaries have to change over

timetime– Cost: Data MovementCost: Data Movement

Goal: Guarantee load balance at low costGoal: Guarantee load balance at low cost– Assumption: Load balance beneficial !!Assumption: Load balance beneficial !!

ContributionContribution– Online balancing -- self-tuning systemOnline balancing -- self-tuning system– Slows down updates by small constant factorSlows down updates by small constant factor

44

RoadmapRoadmap

Model and DefinitionsModel and Definitions Load Balancing OperationsLoad Balancing Operations The AlgorithmsThe Algorithms Extension to P2P SettingExtension to P2P Setting Experimental ResultsExperimental Results

55

Model and Definitions (1)Model and Definitions (1)

Nodes maintain range partition (on a Nodes maintain range partition (on a key)key)– Load of a node = # tuples in its partitionLoad of a node = # tuples in its partition– Load imbalance Load imbalance σσ = Largest = Largest

load/Smallest loadload/Smallest load Arbitrary sequence of tuple inserts Arbitrary sequence of tuple inserts

and deletesand deletes– Queries not relevantQueries not relevant– Automatically directed to relevant nodeAutomatically directed to relevant node

66

Model and Definitions (2)Model and Definitions (2)

After each insert/delete:After each insert/delete:– Potentially fix “imbalance” by modifying Potentially fix “imbalance” by modifying

partitioningpartitioning– Cost= # tuples movedCost= # tuples moved

Assume no inserts/deletes during Assume no inserts/deletes during balancingbalancing– Non-critical simplificationNon-critical simplification

Goal: Goal: σσ < < constantconstant always always– ConstantConstant amortized cost per insert/delete amortized cost per insert/delete – Implication: Faster queries, slower updatesImplication: Faster queries, slower updates

77

Load Balancing Operations Load Balancing Operations (1)(1)

NbrAdjust: Transfer data between NbrAdjust: Transfer data between “neighbors’’“neighbors’’

[0,50)

A B

[50,100)

[0,35) [35,100)

88

Is NbrAdjust good enough?Is NbrAdjust good enough?

Can be highly inefficientCan be highly inefficient ((nn) amortized cost per insert/delete ( ) amortized cost per insert/delete ( n=#nodes n=#nodes ))

A EDCB F

99

Load Balancing Operations Load Balancing Operations (2)(2)

Reorder: Hand over data to neighbor Reorder: Hand over data to neighbor and split load of some other nodeand split load of some other node

A EDCB F

[0,10)

[10,20) [20,30)

[30,40) [40,50)

[50,60)

[40,60)

[0,5) [5,10)

1010

RoadmapRoadmap

Model and DefinitionsModel and Definitions Load Balancing OperationsLoad Balancing Operations The AlgorithmsThe Algorithms Experimental ResultsExperimental Results Extension to P2P SettingExtension to P2P Setting

1111

The Doubling AlgorithmThe Doubling Algorithm

Geometrically divide loads Geometrically divide loads into levelsinto levels– Level i Level i Load in ( 2 Load in ( 2ii,2,2i+1 i+1 ]]– Will try balancing on level Will try balancing on level

changechange Two InvariantsTwo Invariants

– Neighbors tightly balancedNeighbors tightly balanced Max 1 level apartMax 1 level apart

– All nodes within 3 levelsAll nodes within 3 levels Guarantees Guarantees σσ ≤ 8 ≤ 8

1

42

8

2i2i+1

2i+2

Level i

Level 0

Level 2Level 1

Load Scale

1212

The Doubling Algorithm (2)The Doubling Algorithm (2)

A EDCB F

1313

The Doubling Algorithm (2)The Doubling Algorithm (2)

A EDCB F

1414

The Doubling Algorithm (2)The Doubling Algorithm (2)

A EDCB F

1515

The Doubling Algorithm: The Doubling Algorithm: Case 2Case 2

A EDCB F

Search for a blue nodeSearch for a blue node– If none, do nothing!If none, do nothing!

1616

The Doubling Algorithm: The Doubling Algorithm: Case 2Case 2

A DCEB F

Search for a blue nodeSearch for a blue node– If none, do nothing!If none, do nothing!

1717

The Doubling Algorithm (3)The Doubling Algorithm (3)

Similar operations when load goes Similar operations when load goes down a leveldown a level– Try balancing with neighborTry balancing with neighbor– Otherwise, find a red node and reorder Otherwise, find a red node and reorder

yourselfyourself Costs and GuaranteesCosts and Guarantees

– σσ ≤ 8 ≤ 8– Constant amortized cost per Constant amortized cost per

insert/deleteinsert/delete

1818

From Doubling to FibbingFrom Doubling to Fibbing

Change thresholds to Fibonacci numbersChange thresholds to Fibonacci numbers– σσ ≤ ≤ 3 3 4.24.2– Can also use other geometric sequencesCan also use other geometric sequences– Costs are still constantCosts are still constant

FiFi+1Fi+2 = +

1919

More GeneralizationsMore Generalizations

Improve Improve σσ to (1+ to (1+) for any ) for any >0 >0 [BG04][BG04]– Generalize neighbors to c-neighborsGeneralize neighbors to c-neighbors– Still constant cost O(1/ Still constant cost O(1/ ))

Dealing with concurrent Dealing with concurrent inserts/deletesinserts/deletes– Allow multiple balancing actions in Allow multiple balancing actions in

parallelparallel– Paper claims it is okPaper claims it is ok

2020

Application to P2P SystemsApplication to P2P Systems

Goal: Construct P2P system supporting Goal: Construct P2P system supporting efficient range queriesefficient range queries– Provide asymptotic performance Provide asymptotic performance a la a la DHTsDHTs

What is a P2P system? A parallel DB withWhat is a P2P system? A parallel DB with– Nodes joining and leaving at willNodes joining and leaving at will– No centralized componentsNo centralized components– Limited communication primitivesLimited communication primitives

Enhance load-balancing algorithms to Enhance load-balancing algorithms to – Allow dynamic node joins/leavesAllow dynamic node joins/leaves– Decentralize implementationDecentralize implementation

2121

ExperimentsExperiments Goal: Study cost of balancing for different Goal: Study cost of balancing for different

workloadsworkloads– Compare to periodic re-balancing algorithms (Paper)Compare to periodic re-balancing algorithms (Paper)– Trade-off between cost and imbalance ratio (Paper)Trade-off between cost and imbalance ratio (Paper)

Results presented on Fibbing Algorithm (n=256)Results presented on Fibbing Algorithm (n=256) Three-phase WorkloadThree-phase Workload

– (1) Inserts (2) Alternating inserts and deletes (3) Deletes(1) Inserts (2) Alternating inserts and deletes (3) Deletes Workload 1: ZipfWorkload 1: Zipf

– Random draws from Zipf-like distributionRandom draws from Zipf-like distribution Workload 2: HotSpotWorkload 2: HotSpot

– Think key=timestampThink key=timestamp Workload 3: ShearStressWorkload 3: ShearStress

– Insert at most-loaded, delete from least-loadedInsert at most-loaded, delete from least-loaded

2222

Load Imbalance (Zipf)Load Imbalance (Zipf)

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

0 500 1000 1500 2000 2500 3000

Time (x1000)

Lo

ad I

mb

alan

ce

Growing Phase Steady Phase Shrinking Phase

2323

Load Imbalance Load Imbalance (ShearStress)(ShearStress)

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

0 500 1000 1500 2000 2500 3000

Time (x1000)

Lo

ad

Im

ba

lan

ce

Growing Phase Steady Phase Shrinking Phase

2424

Cost of Load BalancingCost of Load Balancing

0

1000

2000

3000

4000

5000

6000

0 500 1000 1500 2000 2500 3000

Time (x1000)

Cu

mu

lati

ve

Co

st

(x1

00

0)

Growing Phase Steady Phase Shrinking Phase

2525

Related WorkRelated Work

Karger & Ruhl [SPAA 04]Karger & Ruhl [SPAA 04]– Dynamic model, weaker guaranteesDynamic model, weaker guarantees

Load balancing in DBsLoad balancing in DBs– Partitioning static relations, e.g., Partitioning static relations, e.g.,

[GD92,RZML02, SMR00][GD92,RZML02, SMR00]– Migrating fragments across disks, e.g., Migrating fragments across disks, e.g.,

[SWZ93][SWZ93]– Intra-node data structures, e.g., Intra-node data structures, e.g.,

[LKOTM00][LKOTM00] Litwin et al. SDDSLitwin et al. SDDS

2626

ConclusionsConclusions

Indeed possible to maintain well-balanced Indeed possible to maintain well-balanced range partitionsrange partitions– Range partitions competitive with hashingRange partitions competitive with hashing

Generalize to more complex load functionsGeneralize to more complex load functions– Allow tuples to have dynamic weightsAllow tuples to have dynamic weights– Change load definition in algorithms!Change load definition in algorithms!**

– Range partitioning is powerfulRange partitioning is powerful Enables P2P system supporting range queriesEnables P2P system supporting range queries

– Generalizes DHTs with same asymptotic Generalizes DHTs with same asymptotic guaranteesguarantees

*Lots of caveats apply. Need load to be evenly divisible. No guarantees offered on costs. This offer not valid with any other offers. Etc, etc. etc.