Spectral Graph Theory, Linear Solvers, and...

IntroductionGraph Based Methods

Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems

Spectral RoundingOpen Questions

Spectral Graph Theory,Linear Solvers, and

Applications

Gary Miller

Carnegie Mellon Universityjoiny work with Yiannis Koutis and David Tolliver

Theory and Practice of Computational LearningJune 9, 2009

Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications




Linear Systems

3x 2y −z = 32x −5y 4z = 7−x 1/2y 2z = 2

Fundamental ConstraintSystem





Matrix Form

3 2 −12 −5 4

−1 1/2 2

xyz

=

372





Solving the General Case

• Dense Case: O(n3) or O(n2.81) Strassen.

• Sparse Case: Still O(n2.81).





Solving the General Case

• Dense Case: O(n3) or O(n2.81) Strassen.• Sparse Case: Still O(n2.81).





An Easy Case

• Upper and Lower Triangular Systems

• O(m) time where m = number of nonzerosentries.

• Goal: Find more easy cases.





An Easy Case

• Upper and Lower Triangular Systems• O(m) time where m = number of nonzeros

entries.

• Goal: Find more easy cases.





An Easy Case

• Upper and Lower Triangular Systems• O(m) time where m = number of nonzeros

entries.• Goal: Find more easy cases.





Symmetric Matrices

• Assume A is symmetric, A = AT .

• Assume A is positive definite, xTAx > 0 forx 6= 0





Symmetric Matrices

• Assume A is symmetric, A = AT .• Assume A is positive definite, xTAx > 0 forx 6= 0





Direct MethodsGaussian Elimination Matrices

• Goal: algorithms that minimize work andspace.

• Trick: View nonzero entries as an undirectedgraph and view pivoting as a graphoperation.





Direct MethodsGaussian Elimination Matrices

• Goal: algorithms that minimize work andspace.

• Trick: View nonzero entries as an undirectedgraph and view pivoting as a graphoperation.





Pivoting

Viewing pivoting as a graph operation.

• Let G = (V, E) be a graph and v a vertex.

• PIV OT (v) :• Make a clique out of neighbors of v.• Remove v.

• Fill: New edges formed.• Work: All edges “touched”.





Pivoting


• Let G = (V, E) be a graph and v a vertex.• PIV OT (v) :

• Make a clique out of neighbors of v.• Remove v.






Pivoting




• Fill: New edges formed.

• Work: All edges “touched”.





Pivoting









Good Pivot Strategies

1970s and 1980s• Planar systems: O(n3/2) work and O(n log n)

fill/space.

• 3D Systems: O(n2) work and O(n3/2)fill/space.

• O(n3/2) space is a problem for ML sizeproblems.







fill/space.• 3D Systems: O(n2) work and O(n3/2)

fill/space.

• O(n3/2) space is a problem for ML sizeproblems.







fill/space.• 3D Systems: O(n2) work and O(n3/2)

fill/space.• O(n3/2) space is a problem for ML size

problems.





Pure Iterative Methods

Solving Ax = b.• Basic method: x(i+1) = (I − A)x(i) + b

• Convergence/Rate is determined by ||I −A||.• Accelerated Methods: Chebyshev Iteration,

Conjugate Gradient.• CG is still too slow.







• Convergence/Rate is determined by ||I −A||.

• Accelerated Methods: Chebyshev Iteration,Conjugate Gradient.

• CG is still too slow.








Conjugate Gradient.

• CG is still too slow.








Conjugate Gradient.• CG is still too slow.





Preconditioned Iterative Methods

Solving B−1Ax = B−1b = b′.• Basic method: x(i+1) = (I −B−1A)x(i) + b′

• Computing the term z = B−1Ax(i)

• y = Ax(i) Forward Multiply• Bz = y Solve the preconditioner system

• Goal: Minimize the number of iteration whileminimizing the cost of the solve.























Classic Preconditioners

• Jacobi: B = Diagonal(A).

• Gauss-Seidel: B = UpperTriangular(A).• SSOR: B = (L + 1

ωD) 1ωD(L + 1

ωD)

• Still too slow and unreliable.






• Jacobi: B = Diagonal(A).• Gauss-Seidel: B = UpperTriangular(A).

• SSOR: B = (L + 1ωD) 1

ωD(L + 1ωD)







• Jacobi: B = Diagonal(A).• Gauss-Seidel: B = UpperTriangular(A).• SSOR: B = (L + 1

ωD) 1ωD(L + 1

ωD)







• Jacobi: B = Diagonal(A).• Gauss-Seidel: B = UpperTriangular(A).• SSOR: B = (L + 1

ωD) 1ωD(L + 1

ωD)






Graph LaplaciansApplications

Graph Laplacian• G = (V,E, w) weighted undirected graph, wij > 0.

• Weighted incidence matrix:

Aij ={

wij if eij ∈ E0 otherwise

• Degree of vi: di =∑

j wij

•

D =

d1 0. . .

0 dn

• Laplacian: L = D −A






Graph Laplacian• G = (V,E, w) weighted undirected graph, wij > 0.• Weighted incidence matrix:

Aij ={



j wij

•

D =

d1 0. . .

0 dn








Aij ={



j wij

•

D =

d1 0. . .

0 dn








Aij ={



j wij

•

D =

d1 0. . .

0 dn








Aij ={



j wij

•

D =

d1 0. . .

0 dn







Classic Applications of the Laplacian

• View each edge a conductor with conductance wij .

• Let V be a column vector of voltages• If LV = c the c residual current needed to maintain the

given voltages.







• View each edge a conductor with conductance wij .• Let V be a column vector of voltages

• If LV = c the c residual current needed to maintain thegiven voltages.







• View each edge a conductor with conductance wij .• Let V be a column vector of voltages• If LV = c the c residual current needed to maintain the

given voltages.






Graph Laplacian’s and the HeatEquations






Graph Laplacian’s and RandomWalks

Transition Matrix: D−1LG

Fundamental Eigenvectors: O(n + m) (Spielman Teng)Trick: Inverse Powering only requires O(log n) iterations.






Laplacian’s and Spring Mass Systems

• G = (V, E, w) weighted graph and wij isviewed a spring constant.

• M is a diagonal matrix of mass constants• Fact: Modes of vibration of Spring-Mass

system G, M are:Eigen-pairs of LGx = λMx.








• M is a diagonal matrix of mass constants

• Fact: Modes of vibration of Spring-Masssystem G, M are:Eigen-pairs of LGx = λMx.








• M is a diagonal matrix of mass constants• Fact: Modes of vibration of Spring-Mass

system G, M are:Eigen-pairs of LGx = λMx.






Spring Mass System






Graph Laplacian’s and MaximumFlow

Graph Maximum Flow: O((m + n)3/2) (Daitch Spielman)






Graph Laplacian’s and ConvexProgramming

Nonuniform TV Denoising: O((m + n)3/2) (Koutis M SinopTolliver)





Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times

Spanning Tree Preconditioners• Vaidya ’93: Use Maximum Weight Spanning

Tree (MST) plus a few edges.

b

c

a

d

e f g

h

j

i






Spanning Tree Preconditioners• Advantages: Easy to find and Easy to solve

their systems.

b

c

a

d

e f g

h

j

i






Spanning Tree Preconditioners• Problem: Small edge weights differences

can make MST bad.

b

c

a

d

e f g

h

j

i






Low Stretch Spanning Trees• EEST ’05: Use low stretch spanning trees

plus a few edges.

b

c

a

d

e f g

h

j

i






Low Stretch Spanning Trees• Advantages: Better condition number.

b

c

a

d

e f g

h

j

i






Low Stretch Spanning Trees• Problem: Super linear time to find.

b

c

a

d

e f g

h

j

i






Low Stretch Spanning Trees• Richter-M ’04: There are no good spanning

trees even for square mesh.

b

c

a

d

e f g

h

j

i






Steiner Tree Preconditioners

• Gremban-M ’94: Use Steiner trees.

b

c

a

d

e f g

h

j

i

g

a b

c d

h i je

f







• Advantages: Better condition number forgraphs like square mesh.

b

c

a

d

e f g

h

j

i

g

a b

c d

h i je

f







• Advantages: Good experimental results.

b

c

a

d

e f g

h

j

i

g

a b

c d

h i je

f







• Problem: Hard to construct in general andanalyze.

b

c

a

d

e f g

h

j

i

g

a b

c d

h i je

f






Steiner Forest Preconditioners

• Koutis-M ’07: Use Steiner Forest.

b

c

a

d

e f g

h

j

i

g

a b

c d

h i je

f







• Advantages: Easy to find and works wellwith recursive solvers.

b

c

a

d

e f g

h

j

i

g

a b

c d

h i je

f







• Advantages: Good experimental results.

b

c

a

d

e f g

h

j

i

g

a b

c d

h i je

f







• Problem: Analysis only for planar systems.

b

c

a

d

e f g

h

j

i

g

a b

c d

h i je

f






Recursive Methods

• Vaidya ’93: Idea: Tree have a lot of degree1-2 degree nodes. Pivot on these nodes andthen find a preconditioner for this graph.

d h i jf ga b ec






Recursive Methods

• Vaidya ’93: Idea: Tree have a lot of degree1-2 degree nodes. Pivot on these nodes andthen find a preconditioner for this graph.

The reduced graph.






The Error

• Recurrence: u(i+1) = Gu(i) + b, for usG = I − A.

• Error: e(i) = u(i) − u, where u = Gu + b

• Fact: e(i) = Gie(0)

• We need: limi→∞ Gi = 0






The Error










The Error










The Error










Condition Number

• Ax = λx

• DEF: λ Eigenvalue and x Eigenvector.• Λ(A) = {0 ≤ λ1 ≤ · · ·λn}.• Condition Number: κ(A) = λn/λ1






Condition Number

• Ax = λx

• DEF: λ Eigenvalue and x Eigenvector.

• Λ(A) = {0 ≤ λ1 ≤ · · ·λn}.• Condition Number: κ(A) = λn/λ1






Condition Number

• Ax = λx

• DEF: λ Eigenvalue and x Eigenvector.• Λ(A) = {0 ≤ λ1 ≤ · · ·λn}.

• Condition Number: κ(A) = λn/λ1






Condition Number

• Ax = λx

• DEF: λ Eigenvalue and x Eigenvector.• Λ(A) = {0 ≤ λ1 ≤ · · ·λn}.• Condition Number: κ(A) = λn/λ1






Convergence Rates

• Basic Method: Convergence Rate= O(1/κ(A))

• Conjugate Gradient: O(1/√

κ(A))

• Conjugate Gradient: ≈ 1/diameter(A).






Convergence Rates



κ(A))







Convergence Rates



κ(A))







Generalized Condition Number

• Goal: Bound condition number of B−1A

• Note: B−1Ax = λx iff Ax = λBx

• DEF: λ is a generalized eigenvalue.• Condition Number: κ(B−1A) = λn/λ1


















• DEF: λ is a generalized eigenvalue.

• Condition Number: κ(B−1A) = λn/λ1















The Support

• Positive Semi-Definite: ∀x xTAx ≥ 0

• Support of A by B:σ(A/B) = min{τ : τB − A is PSD}

• Fact: κ(A, B) = σ(A, B) · σ(B, A).






The Support



• Fact: κ(A, B) = σ(A, B) · σ(B, A).






The Support



• Fact: κ(A, B) = σ(A, B) · σ(B, A).






Estimating Support

• G and H are graphs and V (G) = V (H)

• Path Embedding: φ : E(G) then paths(H)s.t. φ(eij) = Vi · · ·Vj.






Estimating Support ExampleEG: G ≡ K4 and H ≡ 4-cycle

Congestion ≡ 3 and Dilation ≡ 2Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications





History of Planar Solvers

• 1950’s O(n2) (Conjugate Gradient)

• 1970’s O(n1.5) (Nested Dissection) (LRT)• 1990’s O(n1.2) (Combinatorial

Preconditioners) (Vaidya)• 2000’s O(n log2 n) (Low stretch spanning

trees) (ST)• 2006’s O(n) (separator based

preconditioners) (KM)







• 1950’s O(n2) (Conjugate Gradient)• 1970’s O(n1.5) (Nested Dissection) (LRT)

• 1990’s O(n1.2) (CombinatorialPreconditioners) (Vaidya)

• 2000’s O(n log2 n) (Low stretch spanningtrees) (ST)

• 2006’s O(n) (separator basedpreconditioners) (KM)







• 1950’s O(n2) (Conjugate Gradient)• 1970’s O(n1.5) (Nested Dissection) (LRT)• 1990’s O(n1.2) (Combinatorial

Preconditioners) (Vaidya)

• 2000’s O(n log2 n) (Low stretch spanningtrees) (ST)










trees) (ST)










trees) (ST)• 2006’s O(n) (separator based

preconditioners) (KM)






General Laplacian Solver

O(n + m) (Spielman Teng)






0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.80

2

4

6

8

10

12

14

16

18

20

Number of Pixels (in millions)

Ru

nn

ing

Tim

e (s

ecs)

Two dimensional images

Our solverMATLAB’s direct solver






0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.90

2

4

6

8

10

12

Number of Pixels (in millions)

Ru

nn

ing

Tim

e (s

eco

nd

s)Three dimensional images

Our solverdirect solver

Out of memory





Symmetric Diagonally DominateSystems

• Goal: Show how to solve SDD using regularLaplacians.

• Let G = (V, E, w) such that ij ∈ E:wij 6= 0• Weighted incidence matrix: A.• Degree of vi: di =

∑j |wij|

• Generalized Laplacian: L = D − A• Note: Every SDD is a Generalized

Laplacian.







• Let G = (V, E, w) such that ij ∈ E:wij 6= 0

• Weighted incidence matrix: A.• Degree of vi: di =

∑j |wij|


Laplacian.







• Let G = (V, E, w) such that ij ∈ E:wij 6= 0• Weighted incidence matrix: A.


j |wij|• Generalized Laplacian: L = D − A• Note: Every SDD is a Generalized

Laplacian.








∑j |wij|


Laplacian.








∑j |wij|

• Generalized Laplacian: L = D − A

• Note: Every SDD is a GeneralizedLaplacian.








∑j |wij|


Laplacian.Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications




Basic Properties: GeneralizedLaplacian

• xTLx =∑wij>0 wij(xi − xj)

2 −∑

wij<0 wij(xi + xj)2

• Thus L is positive semidefinite.• Claim: Rank = n− 1 if G is connected.







2 −∑

wij<0 wij(xi + xj)2

• Thus L is positive semidefinite.

• Claim: Rank = n− 1 if G is connected.







2 −∑

wij<0 wij(xi + xj)2

• Thus L is positive semidefinite.• Claim: Rank = n− 1 if G is connected.





Solving Generalized Laplacian byChange of Variables

• First Idea: Find a change of variables it get aregular Laplacian.

• Note: multiplying the ith column and row by−1 preserves Laplacian.

• This is just Flipping xi and bi to −xi and −bi.





















Generalized Laplacians Example

+1

V1

−1

V2−1V3 2 +1 −1

+1 2 +1−1 +1 2

x1

x2

x3

=

b1

b2

b3





Change of Variables for Laplacians

+1

V1

−1

V2−1V3

+1−1

2 −1 +1−1 2 +1+1 +1 2

−x1

x2

x3

=

−b1

b2

b3





Orientable Generalized Laplacians

• DEF: G = (V, E, w) is orientable is ∃sequence of flips s.t. w > 0.

• DEF: LG is orientable if G is.• Note: Orientability is linear testable, greedy.







• DEF: LG is orientable if G is.

• Note: Orientability is linear testable, greedy.







• DEF: LG is orientable if G is.• Note: Orientability is linear testable, greedy.






• Claim: If G is connected and not orientablethen L is SPD.

• Proof: Suppose xTLx = 0 and x 6= 0

• Pick a spanning tree T of G and orient it andflipping x.

• WLOG: x is the all-ones vector, a contra!,since G still has negative edges.
































Two-Fold Covers• DEF: The 2-fold cover G = (V , W , w) ofG = (V, W, w) is:

• V = {V1, V1, . . . , Vn, Vn}• E: If wij > 0

add edges < Vi, Vj > and < Vi, Vj > withweight wij

• E: If wij < 0add edges < Vi, Vj > and < Vi, Vj > withweight −wij






• V = {V1, V1, . . . , Vn, Vn}

• E: If wij > 0add edges < Vi, Vj > and < Vi, Vj > withweight wij







• V = {V1, V1, . . . , Vn, Vn}• E: If wij > 0








• V = {V1, V1, . . . , Vn, Vn}• E: If wij > 0







Example: Two-Fold Cover

+1

V1

−1

V2−1V3





Two-Fold Covers and SolvingGeneralized Laplacians

• Let L = L(G) and L = G

• Note:

Lx = b then L

(x

−x

)=

(b

−b

)•

L

(xy

)=

(b

−b

)then L(x/2− y/2) = b






• Let L = L(G) and L = G• L is a regular Laplacian which we can solve

quickly.

• Note:

Lx = b then L

(x

−x

)=

(b

−b

)•

L

(xy

)=

(b

−b

)then L(x/2− y/2) = b







• Note:

Lx = b then L

(x

−x

)=

(b

−b

)

•

L

(xy

)=

(b

−b

)then L(x/2− y/2) = b







• Note:

Lx = b then L

(x

−x

)=

(b

−b

)•

L

(xy

)=

(b

−b

)then L(x/2− y/2) = b





SDD systems

• The 2-fold trick can be run without the factorof two in space and time.

• There should be uses of negative weights inrecommendation problems.

• Naive approach does not seem to work right.





SDD systems








SDD systems








Spectral Graph Partitioning• Idea: Pick a few low frequency eigenvectors.

• Use these vectors to embed the graph in Rd





Spectral Graph Partitioning• Idea: Pick a few low frequency eigenvectors.• Use these vectors to embed the graph in Rd





The Blue Sky ProblemShi Malik applied to an image:





The Blue Sky ProblemShi Malik solution:





The Blue Sky ProblemSpectral Rounding applied to Image:





Spectral RoundingEdge Reweighting

Algorithm:• Solve Lf = λ2Df .• Reweight graph edges getting L′ and D′.• Solve L′f = λ2D

′f

• Repeat while λ2 6= 0.





SR: The reweighting scheme

• View D, f , and λ2 as a function of L

• Subject to Lf = λ2Df and fTDf = 1.• We get: ∂λ2

∂eij= (fi − fj)

2 − λ2(f2i + f 2

j )

• Take a “small” step in the direction of thegradient.

• If an edge goes negative set it to zero.





Medical Examples of SR

Breast Tumors





Medical Examples of SR

Breast Tumors





• Find fast methods for any SPD system.• Find spectral methods that find better cut by

using more than one eigenvector.• Find solvers that work in the L2 norm.• A implementable solver with near linear time

guarantees.


Date post:	29-Jun-2020
Category:	Documents
Upload:	others
View:	7 times
Download:	1 times

Spectral Graph Theory, Linear Solvers, and...

Documents