15-853 Page1

Separatosr in the Real World Practice

Slides combined fromwww.cs.berkeley.edu/~demmel/cs267_Spr07www.cs.cmu.edu/~guyb/realworld/slidesF03/src/separators1.ppt

15-853 Page2

Edge Separators

7

8

3

4

01

2

5

6

An edge separator : a set of edges E’ ½ E which partitions V into V1 and V2

Criteria:|V1|, |V2| balanced

|E’| is small

V1

V2

E’

15-853 Page3

Vertex Separators

7

8

3

4

01

2

5

6

An vertex separator : a set of vertices V’ ½ V which partitions V into V1 and V2

Criteria:|V1|, |V2| balanced

|V’| is small

V1

V2

15-853 Page4

Other names

Sometimes referred to as – graph partitioning (probably more

common than “graph separators”)– graph bisectors– graph bifurcators– balanced graph cuts

Application

15-853 Page5

Telephone network design– Original application, algorithm due to Kernighan

Load Balancing while Minimizing CommunicationSparse Matrix times Vector Multiplication

– Solving PDEs– N = {1,…,n}, (j,k) in E if A(j,k) nonzero, – WN(j) = #nonzeros in row j, WE(j,k) = 1

VLSI Layout– N = {units on chip}, E = {wires}, WE(j,k) = wire

lengthSparse Gaussian Elimination

– Used to reorder rows and columns to increase parallelism, and to decrease “fill-in”

Data mining and clusteringPhysical Mapping of DNA

15-853 Page6

Recursive Separation

78

3

4

01

2

56

8

12

56

7

3

4

0

2

68

1

574

3

0

3 0 7 4

85

1 8 5 2 6

15-853 Page7

What graphs have small separators

Planar graphs: O(n1/2) vertex separators2d meshes, constant genus, excluded minors

Almost planar graphs:the internet, power networks, road networks

Circuitsneed to be laid out without too many crossings

Social network graphs:phone-call graphs, link structure of the web, citation graphs, “friends graphs”

3d-grids and meshes: O(n1/2)

15-853 Page8

What graphs don’t have small separatos

Expanders: Can someone give a quick proof that they don’t have small separators

Hypercubes: O(n) edge separators O(n/(log n)1/2) vertex separators

Butterfly networks:O(n/log n) separators ?

It is exactly the fact that they don’t have small separators that make them useful.

15-853 Page9

Applications of Separators

Circuit Layout (dates back to the 60s)VLSI layoutSolving linear systems (nested dissection)

n3/2 time for planar graphsPartitioning for parallel algorithmsApproximations to certain NP hard problems

TSP, maximum-independent-setClustering and machine learningMachine vision

15-853 Page10

More Applications of Separators

Out of core algorithmsRegister allocationShortest PathsGraph compressionGraph embeddings

15-853 Page11

Available Software

METIS: U. MinnessotaPARTY: University of PaderbornCHACO: Sandia national labsJOSTLE: U. GeenwichSCOTCH: U. BordeauxGNU: Popinet

Benchmarks:• Graph Partitioning Archive

15-853 Page12

Different Balance Criteria

Bisectors: 50/50Constant fraction cuts: e.g. 1/3, 2/3Trading off cut size for balance:

min cut criteria:

min quotient separator:

All versions are NP-hard

21'

'min

VV

V

VV

),min(

'min

21' VV

V

VV

15-853 Page13

Other Variants of Separators

k-Partitioning:Might be done with recursive partitioning, but direct solution can give better answers.

Weighted:Weights on edges (cut size), vertices (balance)

Hypergraphs:Each edge can have more than 2 end pointscommon in VLSI circuits

Multiconstraint:Trying to balance different values at the same time.

15-853 Page14

Edge vs. Vertex separators

If a class of graphs satisfies an f(n) edge-separator theorem then it satisfies an f(n) vertex-separator.

The other way is not true (unless degree is bounded)

|C| = n/2

15-853 Page15

Separator Trees

78

3

4

01

2

56

8

12

56

7

3

4

0

2

68

1

574

3

0

3 0 7 4

85

1 8 5 2 6

15-853 Page16

Asymptotics

If S is a class of graphs closed under the subgraph relation, then

Definition: S satisfies a f(n) vertex-separator theorem if there are constants < 1 and > 0 so that for every G S there exists a cut set C V, with1. |C| < f(|G|) cut size2. |A| < |G|, |B| < |G| balance

Similar definition for edge separators.

15-853 Page17

Separator Trees

Theorem: For S satisfying an (,) n1- edge separator theorem, we can generate a perfectly balanced separator tree with separator size |C| < k f(|G|).

Proof: by picture |C| = n1-(1 + + 2 + …) = n1-(1/1-)

15-853 Page18

Algorithms

All are either heuristics or approximations– Kernighan-Lin (heuristic)– Planar graph separators

(finds O(n1/2) separators)– Geometric separators

(finds O(n(d-1)/d) separators)– Spectral (finds O(n(d-1)/d) separators)– Flow techniques (give log(n)

approximations)– Multilevel recursive bisection

(heuristic, currently most practical)

04/13/2007 CS267 Guest Lecture 19

Partitioning via Breadth First Search

BFS identifies 3 kinds of edges– Tree Edges - part of T– Horizontal Edges - connect nodes at same level– Interlevel Edges - connect nodes at adjacent levels

No edges connect nodes in levels differing by more than 1 (why?)BFS partioning heuristic

– N = N1 U N2, where • N1 = {nodes at level <= L},

• N2 = {nodes at level > L}

– Choose L so |N1| close to |N2|BFS partition of a 2D Mesh using center as root: N1 = levels 0, 1, 2, 3 N2 = levels 4, 5, 6

root

15-853 Page20

Kernighan-Lin Heuristic

Local heuristic for edge-separators based on “hill climbing”. Will most likely end in a local-minima.

Two versions:Original: takes n2 times per stepFiduccia-Mattheyses: takes n times per step

15-853 Page21

High-level description for both

Start with an initial cut that partitions the vertices into two equal size sets V1 and V2

Want to swap two equal sized sets X A and Y B to reduce the cut size.

Note that finding the optimal subsets X and Y solves the optimal separator problem, so it is NP hard.

We want some heuristic that might help.

A B

YX

C

15-853 Page22

Some Terminology

C(A,B) : the weighted cut between A and B

I(v) : the number of edges incident on v that stay within the partition

E(v) : the number of edges incident on v that go to the other partition

D(v) : E(v) - I(v) D(u,v) : D(u) + D(v) - 2 w(u,v)

the gain for swapping u and v

A

B

Cv

u

15-853 Page23

Kernighan-Lin improvement step

KL(G,A0,B0)

(u A0, v B0 ) put (u,v) in a PQ based on

D(u,v) for k = 1 to |V|/2

(u,v) = max(PQ)(Ak,Bk) = (Ak-1,Bk-1) swap (u,v)delete u and v entries from PQ update D on neighbors (and PQ)

select Ak,Bk with best Ck

Note that can take backward steps (D(u,v) can be negative).

A

B

Cv

u

15-853 Page24

Fiduccia-Mattheyses improvement step

FM(G,A0,B0)

u A0 put u in PQA based on D(u)

v B0 put v in PQB based on D(v)

for k = 1 to |V|/2 u = max(PQA)

put u on B side and update Dv = max(PQb)

put v on A side and update D select Ak,Bk with best Ck

A

B

Cv

u

15-853 Page25

A Bad Example for KL or FMConsider following graph with initial cut given in red.

KL (or FM) will start on one side of grid and flip pairs over moving across the grid until the whole thing is flipped.

The cut will never be better than the original bad cut.

2 2 2

1 1 1

15-853 Page26

Performance in Practice

In general the algorithms do very well at smoothing a cut that is approximately correct.

Used by most separator packages either1. to smooth final results2. to smooth partial results during the

algorithm3. K/L tested on very small graphs (|N|<=360)

and got convergence after 2-4 sweeps

4. For random graphs (of theoretical interest) the probability of convergence in one step appears to drop like 2-|N|/30

27

Introduction to Multilevel Partitioning

If we want to partition G(N,E), but it is too big to do efficiently, what can we do?

– 1) Replace G(N,E) by a coarse approximation Gc(Nc,Ec), and partition Gc instead

– 2) Use partition of Gc to get a rough partitioning of G, and then iteratively improve it

What if Gc still too big?

– Apply same idea recursively

28

Multilevel Partitioning - High Level Algorithm

(N+,N- ) = Multilevel_Partition( N, E ) … recursive partitioning routine returns N+ and N- where N = N+ U N- if |N| is small(1) Partition G = (N,E) directly to get N = N+ U N- Return (N+, N- ) else

(2) Coarsen G to get an approximation Gc = (Nc, Ec)

(3) (Nc+ , Nc- ) = Multilevel_Partition( Nc, Ec )

(4) Expand (Nc+ , Nc- ) to a partition (N+ , N- ) of N(5) Improve the partition ( N+ , N- ) Return ( N+ , N- ) endif

(2,3)

(2,3)

(2,3)

(1)

(4)

(4)

(4)

(5)

(5)

(5)

How do we Coarsen? Expand? Improve?

“V - cycle:”

29

Multilevel Kernighan-LinCoarsen graph and expand partition using

maximal matchingsImprove partition using Kernighan-Lin

30

Maximal Matching: Example

31

Example of Coarsening

32

Coarsening using a maximal matching

1) Construct a maximal matching Em of G(N,E)for all edges e=(j,k) in Em 2) collapse matched nodes into a single one Put node n(e) in Nc

W(n(e)) = W(j) + W(k) … gray statements update node/edge weightsfor all nodes n in N not incident on an edge in Em 3) add unmatched nodes Put n in Nc … do not change W(n)… Now each node r in N is “inside” a unique node n(r) in Nc

… 4) Connect two nodes in Nc if nodes inside them are connected in Efor all edges e=(j,k) in Em

for each other edge e’=(j,r) or (k,r) in E Put edge ee = (n(e),n(r)) in Ec

W(ee) = W(e’) If there are multiple edges connecting two nodes in Nc, collapse them, adding edge weights

33

Expanding a partition of Gc to a partition of G

34

Multilevel Spectral Bisection

Coarsen graph and expand partition using maximal independent sets

Improve partition using Rayleigh Quotient Iteration

35

Example of Coarsening

- encloses domain Dk = node of Nc

36

Coarsening using Maximal Independent Sets… Build “domains” D(k) around each node k in Ni to get nodes in Nc

… Add an edge to Ec whenever it would connect two such domains

Ec = empty set

for all nodes k in Ni

D(k) = ( {k}, empty set ) … first set contains nodes in D(k), second set contains edges in D(k)unmark all edges in Erepeat choose an unmarked edge e = (k,j) from E if exactly one of k and j (say k) is in some D(m) mark e add j and e to D(m) else if k and j are in two different D(m)’s (say D(mi) and D(mj)) mark e add edge (mi, mj) to Ec

else if both k and j are in the same D(m) mark e add e to D(m) else leave e unmarked endifuntil no unmarked edges

37

Expanding a partition of Gc to a partition of G

Need to convert an eigenvector vc of L(Gc) to an approximate eigenvector v of L(G)

Use interpolation:

For each node j in N

if j is also a node in Nc, then

v(j) = vc(j) … use same eigenvector component else

v(j) = average of vc(k) for all neighbors k of j in Nc end ifendif

38

Example: 1D mesh of 9 nodes

39

Improve eigenvector: Rayleigh Quotient Iteration

j = 0

pick starting vector v(0) … from expanding vcrepeat j=j+1

r(j) = vT(j-1) * L(G) * v(j-1) … r(j) = Rayleigh Quotient of v(j-1) … = good approximate eigenvalue

v(j) = (L(G) - r(j)*I)-1 * v(j-1) … expensive to do exactly, so solve approximately … using an iteration called SYMMLQ, … which uses matrix-vector multiply (no surprise) v(j) = v(j) / || v(j) || … normalize v(j) until v(j) converges… Convergence is very fast: cubic

40

Example of convergence for 1D mesh

41

Available Implementations

Multilevel Kernighan/Lin– METIS (www.cs.umn.edu/~metis)– ParMETIS - parallel version

Multilevel Spectral Bisection– S. Barnard and H. Simon, “A fast multilevel

implementation of recursive spectral bisection …”, Proc. 6th SIAM Conf. On Parallel Processing, 1993

– Chaco (www.cs.sandia.gov/CRF/papers_chaco.html)

Hybrids possible – Ex: Using Kernighan/Lin to improve a partition

from spectral bisection

42

Comparison of methods

Compare only methods that use edges, not nodal coordinates – CS267 webpage and KK95a (see below) have other

comparisons

Metrics– Speed of partitioning– Number of edge cuts– Other application dependent metrics

Summary– No one method best– Multi-level Kernighan/Lin fastest by far, comparable to Spectral

in the number of edge cuts– Spectral give much better cuts for some applications

• Ex: image segmentation• See “Normalized Cuts and Image Segmentation” by J. Malik, J. Shi

43

Number of edges cut for a 64-way partition

Graph

1444ELTADD32AUTOBBMATFINAN512LHR10MAP1MEMPLUSSHYY161TORSO

# of Nodes

144649 15606 4960 448695 38744 74752 10672 267241 17758 76480 201142

# of Edges

1074393 45878 94623314611 993481 261120 209093 334931 54196 1520021479989

Description

3D FE Mesh2D FE Mesh32 bit adder3D FE Mesh2D Stiffness M.Lin. Prog.Chem. Eng.Highway Net.Memory circuitNavier-Stokes3D FE Mesh

# Edges cut

88806 2965 675 194436 55753 11388 58784 1388 17894 4365 117997

Expected# cuts

6427 2111 1190 11320 3326 4620 1746 8736 2252 4674 7579

3D mesh

31805 7208 3357 67647 13215 20481 5595 47887 7856 20796 39623

Expected # cuts for 64-way partition of 2D mesh of n nodes

n1/2 + 2*(n/2)1/2 + 4*(n/4)1/2 + … + 32*(n/32)1/2 ~ 17 * n1/2

Expected # cuts for 64-way partition of 3D mesh of n nodes =

n2/3 + 2*(n/2)2/3 + 4*(n/4)2/3 + … + 32*(n/32)2/3 ~ 11.5 * n2/3

For Multilevel Kernighan/Lin, as implemented in METIS (see KK95a)

Thank you!

44