Partitioning Sparse Matrices with Eigenvectors of Graphs
Alex Pothen 1, Horst D. Simon 2, and Kang-Pu Paul Liu
Report RNR-89-009, July 1989
NAS Systems Division
NASA Ames Research Center, Mail Stop T-045-1
Moffett Field, CA 94035
July 25, 1989
1The first and third author are with the Computer Science Department, The
Pennsylvania State University, University Park PA 16802
2The second author is an employee of Boeing Computer Services, Bellevue,
WA 98128
PARTITIONING SPARSE MATRICES WITH EIGENVECTORS OF
GRAPHS
ALEX POTHEN., HOILST SIMONt AND KANG-PU PAUL LIOU$
Abstract. The problem of computing a small vertex separator in a graph arises in the context of
computing a good ordering for the parallel factorization of sparse, symmetric matrices. An algebraic approach
to computing vertex separators is considered in this paper. It is shown that lower bounds on separator sizescan be obtained in terms of the eigenvalues of the Laplacian matrix associated with a graph. The Laplacian
eigenvectors of grid graphs can be computed from Kronecker products involving the eigenvectors of path
graphs, and these eigenvectors can be used to compute good separators in grid graphs. A heuristic algorithmis designed to compute a vertex separator in a general graph by first computing an edge separator in the
graph from an eigenvector of the Laplaeian matrix, and then using a maximum matching in a subgraph to
compute the vertex separator. Results on the quality of the separators computed by the spectral algorithm
are presented, and these are compared with separators obtained from Automatic Nested Dissection and the
Kernighan-Lin algorithm. Finally, we report the time required to compute the Laplacian eigenvector, andconsider the accuracy with which the eigenvector must be computed to obtain good separators. The spectral
algorithm has'the advantage that it can be implemented on a medium size multiprocessor in a straightforward manner.
AMS(MOS) subject classifications: 65F50, 65F05, 65F15, 68R10.
Keywords. graph partitioning, graph spectra, Laplacian matrix, sparse matrix, vertex
separator, parallel reordering algorithms.
1. Introduction. In the solution of large, sparse, positive definite systems on parallel
computers, it is necessary to compute an ordering of the matrix such that it can be factored
efficiently in parallel. For large problems, the storage required for the structure of the
matrix may exceed the storage capacities of a single processor, and this parallel ordering
itself will need to be computed in parallel. One strategy to compute a good parallel order is
to employ the divide and conquer paradigm: Find a set of vertices in the adjacency graph of
the matrix, whose removal disconnects the graph into two nearly equal parts. Number the
vertices in the separator last, and recursively number the vertices in the two parts by the
same strategy. This strategy is employed in the well-known Nested Dissection algorithm for
ordering matrices for sparse factorization.
In computing an ordering by the above approach, at each step, the following Partitioning
problem needs to be solved. Given an adjacency graph G of a sparse matrix, this problem is
to find a vertex separator S such that S has few vertices and S disconnects G \ S into two
parts A, B, with nearly equal numbers of vertices.
* Computer Science Department, The Pennsylvania State University, Whitmore Lab, University Park, PA
16802. Electronic address: [email protected], [email protected]. The research of
this author was supported by National Science Foundation Grant CCR-8701723 and U.S. Air Force Officeof Scientific Research Grant AFOSR-88-0161.
t Numerical Aerodynamic Simulation (NAS) Systems Division, Mail-Stop 258-5, NASA Ames ResearchCenter, Moffett Field, CA 94035. The author is an employee of Boeing Computer Services. Electronic
address: simon_orville.nas.nasa.gov
Computer Science Department, The Pennsylvania State University, Whitmore Lab, University Park, PA16802.
In this paper, we consider a spectral algorithm for solving the partitioning problem. We
associate with the given sparse, symmetric matrix (and its adjacency graph), a matrix called
the Laplacian matrix. We compute a particular eigenvector of the Laplacian matrix, and
use its components to initially partition the vertices into two sets A _, B _. The set of edges
joining A _ and B _ is an edge separator in the graph G. A vertex separator S is computed
from the edge separator by a matching technique.
This spectral algorithm has three features that distinguish it from previous algorithms
that are worthy of comment.
First, previous algorithms for computing separators, such as Nested Dissection or the
Kernighan-Lin algorithm make use of local information in the graph, viz. information about
the neighbors of a vertex, to compute separators. The spectral method employs global
information about the graph, since it computes a separator from eigenvector components.
Thus the spectral method has the potential of finding separators in the graph that are
qualitatively different from the separators obtained by previous approaches.
Second, we can view the spectral method as an approach in which a vertex in the graph
makes a continuous choice, with a weight between +1 and -1, about which part in the initial
partition it is going to belong to. All vertices with weights below the median weight form
one part, and the rest, the other part. In the Kernighan-Lin method, each vertex makes a
discrete choice (zero or one) to belong to one set. The weights in the spectral method can
be used to move a few vertices from one part to the other, if a slightly different partition is
desired in the course of the separator algorithm.
Third, the dominant computation in the spectral method is an eigenvector computation
by a Lanczos or similar algorithm. This distinguishes the new algorithm from standard
graph theoretical algorithms computationally. Most of the computation is based on standard
vector operations on floating point numbers. Becasue of its algebraic nature the algorithm is
parallelizable in a fairly straight forward manner on medium-grain multiprocessors used in
scientific computing. Furthermore, since most of the computations are also vector floating
point operations, this algorithm is well suited for vector supercomputers used for large scale
scientific computing.
This paper is organized as follows. We include background material on the spectral
properties of Laplacian matrices and their relevance to graph partitioning in § 2. We also
review earlier work on computing edge separators from the eigenvectors of the adjacency ma-
trix in this section. In § 3, we obtain lower bounds on the size of the best vertex separators
of a graph in terms of the eigenvalues of the Laplacian matrix. Two different techniques for
proving lower bounds are illustrated: One uses the Courant-Fischer minimax criterion, and
the second employs an inequality from the proof of the Wielandt-Hoffman theorem. _Ve then
show that the spectra of rectangular and square grid graphs can be computed explicitly from
the spectra of path graphs by employing suitable graph products and Kronecker products
in § 4. We proceed to show how good edge and vertex separators in the grid graphs can be
computed from the spectral information. In § 5, we describe our heuristic spectral algorithm
to compute vertex separators in general graphs. The algorithm initially computes an edge
separator, and then uses a maximum matching in a subgraph to compute the vertex sepa-
rator. Results about the quality of the separators computed by the algorithm is presented
in § 6. In this section, we also compare the spectral separators with separators computed
by Automatic Nested Dissection and the Kernighan-Lin algorithm, as well as with results
obtained recently by Liu [31] and Leiserson and Lewis [29]. The time required to compute
the Laplacian eigenvectors with the Lanczos algorithm and the accuracy needed in the eigen-
vector to obtain good separators are addressed in § 7. The final § 8 contains our conclusions
and some directions for future work.
2. Background. Let G : (V,E) be an undirected graph on IvI = - vertices. The
n x n adjacency matrix A = A(G) has element a_.,_ equal to one if (v,w) E E, and zero
otherwise. By convention, a_.,_ is zero, for all v E V. The rows and columns of the matrices
associated with a graph are indexed by the vertices of the graph, their order being arbitrary.
Let d(v) denote the degree of a matrix, and defne D to be the n x n diagonal matrix with
el,,,,, = d(v). The matrix Q = Q(G) = D - A is the Laplacian matriz of G.
Let the edges of the graph G be directed arbitrarily, and let C denote the vertex-edge
incidence matrix of the directed graph. The IV I x IEI matrix C has elements
+1c.,,,e = _-i
if v is the head of e
if v is the tail of e
otherwise.
It is easy to verify [7] that Q(G) = CC t, and that Q is independent of the direction of the
edges in C.
The spectral properties of Q have been studied by several authors. Since
z_tOg = x_CC%_
== Z -
(v,,.)eE
it is easily seen that Q is positive semidefiuite. Let the eigenvalues of Q be ordered
$1 = 0 _<)_2_ ... $..
An eigenvector corresponding to )q is e_, the vector of all ones. The multiplicity of the zero
eigenvalue is equal to the number of connected components of the graph. If G is connected,
then the second smallest eigenvalue Az is positive. We call an eigenvector V_corresponding
to )t2, a second eigenvector.
Fiedler [17, 18] has studied the properties of the second eigenvalue A2 and a corresponding
eigenvector y_. He calls _2 the algebraic connectivity, and relates it to the vertex and edge
cortnectivities of a graph. He has also investigated the partitions of G generated by the
components of the eigertvector y_. One of his results of interest in this paper is the following.
THEOREM 1. Let G be a connected graph, and let y be an eigenvector corresponding to
A2. For a real number r >_ O, define V_(r) = {v E V : y,, > -r}. Then the subgraph induced
by V_(r) is connected. Similarly, for a real number r <__O, let the set V2(r) = {v E V : y_, <
Irt}. The subgraph i,_duced by W2(r) is also co,_nected.3
In both sets V1 and V2, it is necessary to include all vertices with zero components for
the theorem to hold. The role played by these latter vertices in the connectedness of the two
subgraphs has been investigated at greater length by Powers [40, 41].
A corollary to this result is that if W # 0 for all v E V, then the subgraphs induced by
the sets
P = {vEV:v,,>O},
N = {vsV:w<O}
are both connected subgraphs of G.
The eigenvectors of the adjacency matrix corresponding to its algebraically largest eigen-
values have also been used to partition graphs. It is of interest to ask if a similar theorem
holds for an eigenvector corresponding to the second largest eigenvalue of the adjacency
matrix.
Let z_, V__denote eigenvectors corresponding to the Mgebraically largest and second largest
eigenvalues, respectively, of the adjacency matrix of G. By Perron-Frobenius theory, it is
known that all components of z_ are positive. If a is a nonnegative number, then the subgraph
induced by V1 = {v E V : y_ + c_z_ >_ 0} is connected. Similarly, if a is a nonpositive number,
then the subgraph induced by V2 = {v E V :y_ - lalz_ < 0} is also connected.
Mohar [33] has made use of the second Laplacian eigenvector V_to study the isoperimetric
number, i(G). If U is a subset of the vertices of the graph G, and 6U denotes the set of edges
with one endpoint in U and the other in V \ U, then
16uii(G)= minivt_<./2iVl
Mohar proved that
1A
where A is the maximum degree of a vertex. The proof of the upper bound makes use of
the set W = {v E V : y_ > 0}, with the sign of the eigenvector chosen such that W has no
more than n/2 vertices.
The relationship of the Laplacian spectrum to several graph properties have been con-
sidered by several authors; two recent survey articles by Mohar [34] and Bien [6] describe
some of these results.
Spectral methods for computing edge separators have been considered by several re-
searchers.
Donath and Hoffman [12] obtain lower bounds on the size of an edge separator in terms
of the eigenvalues of the matrix A + U, where A is the adjacency matrix, and U is a diagonal
matrix with the property that trace(U) = -2[E[. Barnes and Hoffman [4] consider graphs
with weights on the edges, and obtain lower bounds on the weights of edge separators in terms
of the eigenvectors corresponding to the algebraically largest eigenvalues of a matrix A + D.
Here, A is the weighted adjacency matrix, and D is an n x n matrix such that the elements
of A + D sum to zero. They formulate the partitioning problem as a quadratic integer4
programming problem, and then approximate the latter problem by linear programming
problems, whose solutions yield the lower bounds.
Barnes [3] shows that the problem of partitioning a graph into k > 2 parts such that the
number of edges cut by the partition is minimum is equivalent to a matrix approximation
problem, in which the adjacency matrix is approximated by a partition matrix P. The
n × n matrix P has element Pld equal to one if vertices i and j belong to the same part,
and zero otherwise. He finds a heuristic solution to this approximation problem by solving
a transportation problem involving the eigenvectors corresponding to the k algebraically
largest eigenvalues of A.
Boppana [9] has described a spectral method using the eigenvalues and eigenvectors of
the adjacency matrix which he proves will find a minimum size bisection for almost all graphs
in a certain class of random graphs.
3. Lower Bounds. We obtain lower bounds on the sizes of vertex separators in terms
of the eigenvalues of the Laplacian matrix Q(G) in this section. The lower bounds hold for
any vertex separator in the graph; in particular, these bounds apply to the smallest separator
in the graph. We assume the graph G to be connected.
Let G = (V, E) denote a graph on IVl = n vertices, and let A be a subset of its vertices.
Denote by p(v, A) the distance of a vertex v from A, i.e., the fewest number of edges in a
shortest path from v to a vertex in A. Let S denote the set of vertices which are at a distance
less than p > 2 from A, and not belonging to A. Hence
s = e v \ A: A) < p}.
Define B = V \ (A U S); if B # 0, then the distance between A and B, p(A,B) = p. If
p > 2, the set S is a wide separator that separates A from B. If p = 2, we get the commonly
used notion of separators. Wide separators have been used in sparse matrix algorithms by
George and Ng [21], and Gilbert and Zmijewski [22].
Let EA denote the set of edges with both endpoints in A, and EAs denote the set of
edges with one endpoint in A, and the other in S. The sets EB, Es, and Ess are defined
similarly. In the following, it wiU be convenient to work with the fractional sizes a = IAI/n,
b - IBI/n, and = ISl/n. The degree of a vertex v win be denoted by d(v), and A win
denote the maximum degree of vertices in G.
The first result is a lower bound on the size of a wide separator separating any pair of
vertex disjoint sets A and B which are at a distance p from each other.
TrlEOREM 2. Let A, B be disjoint subsets of vertices of G which are at a distance p > 2
from each other. Let S denote the set of vertices not belonging to A which are at a distance
less than p from A. Then
s 2+_s-p2a(1-a) >0, where/3=(A/A2)+p2a-1.
Proof: Let _e, 0 be the vector of all ones and all zeros, respectively. The Courant-Fischer
minimax principle states that
(I)
_2 = rain z_JQz_
e_'z_=o
= rain E(+,_)_s(z+- zj) _•__0_ E;'=__+
e_tz._=O
Using the Lagrange identity
2 (z+ =j)2,n X i -- X i -: --
i=1 i,j= li<j
Fiedler [18]derived the following inequality, valid for all non-constant n-vectors z__,from (1).
(2) n Z (_'- _)' >-4 Z (_,- _)_.(i,j)EE i,jEV
i<j
We prove the result by making an appropriate choice of z_ in the above inequality.
Choose the v-th component of z_ be
21 rain ir +,- j%+,l
z, = --- tp, p[ v, A )._.P
If v E A, then z_ = 1; if v E B, then z,, = - 1; and if v E S, then - 1 +(2/p) < z,, < 1- (2/p).
Also, if v, w are adjacent vertices, then Ix. - z_, I < 2/p.
The left-hand side of equation (2) has nonzero contributions from three terms, and it
can be bounded from above as follows.
(3)
(+,- ++)_(iS)EE
=l=+ +,,(',j)_ (i,j)_s (id)+E
\iEA,jES iEB,jES iES,jES/
4 (IEAsl + IEssl + IEsl)
4
Similarly, nonzero contributions to the right-hand side of equation (2) also come from
three terms, and we obtain a lower bound as shown.
iS_vi<j
\iE ,jES IEA,jEB iEB,jE$
> i-(I- ) n'as + (l-(-l))'n'ab +
6
( 2)-1-(-1+-) n2bsP
4il 2
-- p2 ((a + 5)8 'k p2a(1 -a- s))
4n2((1- +p2a(1- a - 8)).(4) - p2
Using inequalities (3) and (4) in Fiedler's inequality (2), and canceling common terms,
we obtain
aA >_ A2 ((I -- 8)8 q- p2a(1 -a- s)).
After some rearrangement, this yields the desired result. •
Fiedler [17] proved that A2 satisfies
)_2 _ (n/(n - 1)) rain {d(v) : v • V}.
Mohar [33] proved that for all graphs except the complete graph Kn, A > A2. Thus for all
graphs except the complete graph, the ratio A/)_2 >_ 1, and fl is a positive number. Indeed,
for all the adjacency graphs of sparse matrices that we have computed partitions, the ratio
A/A2, and hence j3, is much larger than one.
COROLLARY 3. If fl > p, then
8 >__p:a(1- _) _ p2_(1-_)/7 (A/_2) + p2a- 1"
Proof: Let sl, 82 be the roots of the quadratic equation corresponding to the inequality
in Theorem 2, with sl < 82. Then s _> 82, and
1 (-/7 + (732 -b 4p_a(1 - a)) 1/2)82_
If/3 __ 2p(a(1 - a)) 1/2, then expanding the rhs in power series yields the result.
It remains to verify the condition of the coronary. Since (a(1 - a)) in has its maximum
value 1/2 when 0 __ a g 1, the power series expansion is valid when 73 __ p. •
The corollary exhibits the dependence of vertex separator sizes on A2: the smaller the
second eigenvalue, the larger the ratio A/A2, and the smaller the lower bound on the vertex
separator size. The corollary also shows the dependence on the lower bound on the distance
p and the fractional size of the set A.
The common situation of a separator corresponds to p = 2. In this case, the quadratic
inequality becomes
s 2+fls-4a(1-a)>_O, with_=(A/A2)+4a-1.
After some simplification, it can be seen that the inequality in Theorem 2.1 of Alon, Galil,
and Milman [11 is equivalent to the above inequality. In this case, when/3 >_ 2, we obtain
the lower bound
s > 4a(1 - a)- (A/4) + 4a -- i"
7
Mohax (Lemma 2.4, [32]) has obtained a lower bound on vertex separators in terms of
the ratio _,,/_2.We can also obtain lower bounds on edge separators. By omitting the last step in
equation (3), and replacing the sum lEAs[ + IEBsl + IEs[ by IEI- IEAI- IEBI, we have
IEI - [EA[ - ]Esl > ),,n ((a + b) - (a + b) 2 + p2ab) ,
for two vertex disjoint sets A, B which axe a distance p apart.
Alon and Milman [2] have obtained the inequality
p2 )_2n p2 )_2nabIEI-IEAI-IEBI> =((i/a) + (lib)) (a + b)"
These techniques can also be used to bound the number of edges separating two disjoint sets
of vertices from each other.
A second lower bound. We now obtain a lower bound that exhibits another factor
influencing the size of vertex separators. The technique used is derived from the Wielandt-
ttoffman theorem, and has been previously used by Donath and Hoffman [12] to obtain lower
bounds on edge separators.
Let S be a vertex separator that separates the graph G into two sets A and B, with
IAI > IBI > ISI. Let d(v) denote the degree of a vertex v, and let i(v) denote the 'internal'
degree of v, i.e., the number of edges incident on v with the other endpoint in the same set
aS _.
Recall that the eigenvalues of the Laplacian matrix Q are ordered as
_1 = 0 < _2 _< _3... <_ _n.
Let the n × n matrix J = diag(Ja, Jb, Jc), where J_ is the na × na matrix of all ones, and Jr,,
Jc are similarly defined. The eigenvalues of J are
_tl = na > tz2 = nb > tz3 = ns > l_4 .... #. =0.
THEOREM 4. Let S be a vertex separator that divides a graph G into two parts A, B,
with IAI > IB[ _> IS[. Then
s > (1 - a))_2- 2A -- ($z --)_:)"
Proof: From the proof of the Wielandt-Hoffman theorem [26] (see also [121),
n
trace(-OJ) _< __.(-)_,1#,.i=1
Ca.nceling out the minus signs yields
n
(5) trace(Q J) _ _ A,#_.i=1
8
We now compute both sides of the above inequality.
The right-hand side is
(6)
n
i=I
= na.O+nb.)_2+ns.)t3
= nbX2 +nsX3
= n(1 - a- s)X2 + nsXs.
To evaluate the left-hand side, we partition the symmetric matrix Q to conform to J:
Q Q.. OQ..)o Q_ Qb. •
Qi, Q_. Q-
(7)
trace(Q J) = trace(QooJo) + trace(Q_Jb) + trace(Q.J. )
(_ + _-_ + _) d(v) - i(v)yEA yES yES
- 2(IEI- IEAI- IEBI- IEsl)_< 2(IEI- IEAI- IEBI)= 2nsA.
Substituting the inequalities (6) and (7) in (5), we obtain
2nsA > n(1 -- a-- s)X2 + nsXs.
After some rearrangement this yields the final result. •
This last lower bound on a vertex separator size shows as before that the magnitude of
A2 influences the lower bound; it also shows that the 'gap' between Xz and A2 has an effect.
The bounds are large when the gap is large.
Donath and Hoffman [12] obtained lower bounds on the number of edges cut by the
partition when the set of vertices is partitioned into k sets, in terms of the eigenvalues of a
matrix M = A + U, where A is the adjacency matrix and U is a diagonal matrix constrained
only by the requirement that its diagonal elements sum to -2[El.
Finally, a word of caution is in order about these lower bounds. These bounds should
be considered the same way one treats an upper bound on the error in an a priori roundoff
error analysis [44] (cited in [23]). The lower bounds obtained are not likely to be tight,
except for certain classes of graphs. They do illustrate, however, that a large ),2, with an
accompanying small A/A2, will result in large sizes for the best separators in a graph.
4. Partitions of grid graphs. In this section we show that the second eigenvector of
the Laplacian matrix can be used to find good edge and vertex separators in grid graphs. The
vertex separators obtained will be identical to the separators at the first level in a theoretical
nested dissection (ND) scheme.
0.3
0.2
0.I
8
_ o
•_ .0.1
-0.2
.0.30
Fic_ile_ vec_r of the pal:h
10 1 20 25 30
verlex number
FIG. 1. The second Laplacian eigenveetor of the path.
To compute separators by this technique, we need to obtain the eigenvectors of grids.
We will show that the Laplacian spectra of grid graphs can be explicitly computed in terms
of the Laplacian spectra of path graphs. Thus we can compute the second eigenvectors of
the grid graphs in terms of the Laplacian eigenvectors of the path graph.
The path graph. Let P,_ denote the path graph on n vertices. For concreteness, we
assume in the discussion to follow that n >_ 2 is even. We number the vertices of the path
from 1 to n in the natural order from left to right. Let Cn --- _r/n. We denote the elements
of a vector z_ by writing its i-th component as (z_).
LEMMA 5. The Laplacian spectrum of P,, is
1
= 4 sin 2 (_(k - 1)¢.),
= ( cos((/- 1/2)(k- 1)¢n) ), fork= 1,...,n, i= 1,...,n.
As k ranges from 1 to n, the angle (1/2)(k- 1)¢. varies from zero to a'/2; hence
the eigenvalues are ordered as )h,. < _,,,... < _,_,,_. Note that )_1,. = 0, z_l,,, = !, and
A2,- = 4 sin 2 (_), and z_.2,,_= ( cos((/- 1/2)¢,) ). The components of !_2., corresponding to
vertices of P30 are shown in Fig. 1.
Let zt denote the median (n/2-th largest) component of the second eigenvector, and
partition the vertices of the path into two sets, one set consisting of all vertices with com-
ponents less than or equal to the median component, and the other consisting of all vertices
with components larger than the median component. This partitions the path into equal
size subsets of vertices, one consisting of the vertices with positive eigenvector components
and the other consisting of vertices with negative components.
If z, denotes the component of the second eigenvector corresponding to vertex v, ]z,-z_,]
is a measure of the distance between the vertices v and w in the path: Fiedler calls this the
10
algebraic distance between v and w.
Graph and Kronecker products. We can compute the spectra of grid graphs from
the spectra of the path graph. We require the concepts of graph products and the Kro-
necker products of matrices. Our notation for graph products is from Cvetkovic, Doob, and
Sachs [11], and a good discussion of Kronecker products may be found in Fiedler [19].
For i = 1, 2, let Gi = (Vi, E/) be graphs. The Cartesian sum G1 + G2 is the graph
(V_ x V2, E), where vertices (il,jl) and (i2,j2) are joined by an edge if either il = i2 and
{jl,j2} is an edge in G2, or jx = j2 and {ix,i2} is an edge in G_.
The Cartesian product G_ "G2 is the graph (V_ × V2,F) where vertices (il,jl) and (i_,j2)
are joined by an edge if {i_,i2} is an edge in G_ and {jl,j2} is an edge in G2.
The strong sum G1 • G2 is the graph (V1 × V2, E U F); thus it contains the edges in both
the Cartesian sum and the Cartesian product.
It is easy to verify that the Cartesian sum Pn + P,,_ is the five-point m × r/grid graph,
and that the strong sum P,, @ P,_ is the nine-point rn × n grid graph.
Since th.e grid graphs can be obtained from appropriate graph products of the path
graph, the Laplacian matrices of the rrt × n grid graphs can be obtained from Kronecker
products involving the Laplacian matrices of the path graph. If C is a p × q matrix, and D is
r × s, the Kronecker product C ® D is the pr × qs matrix with each element dij of D replaced
by the submatrix ( Cdij ). (Equivalently we can consider each element caj of C replaced by
the submatrix ( cajD ).)
The five-point grid. We consider the rrt x n five-point grid, and without loss of generality
consider rrt < n. For concreteness, initially we consider the case when n is even, and rrt < n.
At the end of this section, we discuss the case when rt is odds or rrt = ft. We draw the rrt x n
grid with n vertices in each row and rn vertices in each column.
Let Q denote the Laplacian matrix of the five-point rrt x n grid graph, R., denote the
Laplacian matrix of the path graph on n vertices, and 1,, be the identity matrix of order n.
Recall that )_k,n, z__,n denotes the k-th eigenpair (when eigenvalues are listed in increasing
order) of the path graph with n vertices. The following result is well-known; we include a
proof for completeness, and because we wish to indicate how a similar result is obtained for
the Laplacian spectrum of a modified nine-point grid.
Trlv.ortEM 6. The Laplacian spectrum of the rrt x n five-point grid is
tZlc.l = )ttc,n + .Xl,m,
_.t = Z__k.,,®_,,_, k=l,...,n, l=l,...,m.
Proof: It is easy to verify that the Laplacian matrix of the five-point grid can be expressed
in terms of the Laplacian matrix of the path graph as
Q= P_®I,_ + I.®P_.
The first term in the sum creates m copies of the path on n vertices, and the second term
adds the 'vertical' edges which join neighboring vertices in each column of the grid.
11
We show that Ak,t, z_k,z is an eigenpair of Q.
Qz_,,_ ® _,., = (e_®I_.)(_,.®_,,m) + (I.®e_)(_,.®_,,_.)= (e_,.) ® (I._,,._) + (I._,.) ® (_,,_)= Ak,.z__,. ® _,,,. + z__,. ® Az,,_,m
= (Ak,. + A+,.,)_,,, ® z,,,_.
The transformation from the first llne to the second line uses the associativity of the Kro-
necker product. •
The smallest eigenvalue #1,1 = $1,., + $1,,_ is zero, since Q is positive semidefmite. The
next smallest eigenvalue is either #2,1 or #1,2. Since
#2,1 = As,. + Al,.,, = 4 sin2(_),
gl,2 = At,.+ A2,,,. = 4 sin2(_),
and
and m < n, the second smallest eigenvalue is/z2,1. The corresponding eigenvector is
y_2,1= _..® a_,..= (cos(i- I/2)¢.)® !.
The components of the _2,1 are constant along each column of m vertices, and the
components decrease from left to right across a row. Columns numbered 1 to n/2 have
positive components, and the rest of the columns have negative components. The second
eigenvector components for the m × n five-point grid are shown in Figure 2.
These results show that the second eigenvector of the grid can be used to compute
good edge separators and vertex separators. Let y_ denote this eigenvector in the following
discussion, and let Yt denote the median component ((mn/2)-th largest component out of
ran). From the preceding discussion, yt corresponds to the components of the vertices on
the n/2-th column. Let y_ denote the eigenvector component corresponding to vertex v.
COROLLARY 7. Let V denote the set of vertices of the five-point m × n grid (m < n, n
even}, and let V be partitioned by its second eigenvector as follows:
A'={v:y,,<y,}, B'=V\A.
If E' denotes the set of edges joining A' to B', then E' is an edge separator of size rn which
separates the grid into two parts each with (ran�2) vertices. Further, if S denotes the set of
endpoints of E' which belong to B', then S is a vertex separator of size m which separates
the grid into two parts of (ran�2) and m((n/2)- 1) vertices.
The Corollary follows from noting that A' consists of vertices in the columns 1 to n/2
of the grid, and B' is the remaining set of columns. The edge separator E' consists of the m
edges of the grid which join vertices in column n/2 to column (n/2) + 1. Finally, the vertex
separator S consists of vertices in column (n/2) + 1. Note that the vertex separator is the
same as the separator at the first level found by theoretical nested dissection. Buser [10] has
shown that the edge separator E' yields the optimal isoperimetric number for grid graphs.12
The secondadjacencycigcnvcctorofthegrid.
FIG. 2. The second Laplacian eigenvectorof the._ve-pointgrid.
We now consider the two special cases. One corresponds to the number of columns
r_ being odd. Then vertices in the middle column ((n -F 1)/2-th column) have eigenvector
components equal to zero; columns numbered smaller than the middle column have positive
components, and columns numbered higher have negative components. Again, across each
row, the components decrease from left to right.
The second case corresponds to a square grid, rn = n. Then /_2,1 = l_1,2, and the
second smallest eigenvalue of Q has geometric multiplicity two. The two linearly independent
eigenvectors obtained by the graph product approach are
Y---2,1= x._._,,_® xi,.
Y-1,2 = x-l, '_ ® x._,..
The eigenvector _,i has components as described earlier for the rectangular case, withcomponents constant along each column, and the components decreasing from positive to
negative values across each row. If rt is odd, the middle column has its components equal
to zero. The eigenvector Y-1.2 has components constant across each row, and decreasing from
bottom to top along each column. If n is odd, the middle row has components zero, and
rows below it have positive components, and rows above have negative components. If rt is
even, rows one to n/2 have negative components, and the rest have positive components.
From these two independent eigenvectors, we obtain a middle column and a middle row as
the vertex separators.
The nine-point grid. Let Q' denote the Laplacian matrix of the nine-point grid, and let13
D,* be the rt × n diagonal degree matrix of the n-vertex path. As before, let R._ denote the
Laplacian matrix of the n-vertex path, and/,* the identity matrix of order n. It is again not
difficult to verify that
Q'= t_ ®I,_+ I,*@t_ + R,_®D,,, + D,*®R,,,-P_®R_.
Unfortunately, the spectrum of Q' can not be expressed in terms of the spectra of the path
graphs, as for the five-point grid.
However, we can first embed the nine-point grid graph in a modified grid, whose Lapla-
clan spectrum is computable in terms of the spectra of the path graphs, and then partition
the modified grid. We use the partition of the modified grid to partition the nine-point grid.
The necessary modification to the nine-point grid is as follows. Replace each boundary
edge of the m x n grid by two edges joining the same endpoints. Let Q denote the Laplacian
of the resulting multigraph.
THEOREM 8. The spectrum of Q is
ttk,t = 3($k,,* + )q,..) - Sk,.._z,-_
y__,t = z__,,*®_,,_, fork= 1,...,n, l= l,...,m.
Proof: It is easy to show that
Q = 3(P_ ® I,_ + I,* ® P_) - P_ ® P_.
A direct computation, as in Theorem 6, shows that /zk,t, Y-_,Zis an eigenpair of Q. •
Note that the eigenvectors of the modified nine-point grid are the same as the eigen-
vectors of the five-point grid. As in the five-point grid, the smallest eigenvalue #1,1 is zero;
the second smallest eigenvalue is #_,1 = 12 sin2(¢,*/2). The eigenvector _,1 is the second
eigenvector of the five-point grid. Hence the partitions of the modified nine-point grid are
exactly the same as those of the five-point grid.
Adjacency spectra of paths and grids. The adjacency spectra of grid graphs can
also be computed in terms of the adjacency spectrum of the path. Let A,* denote the
adjacency matrix of the path on n vertices, P,*, and denote a,* - _r/(n + 1). The adjacency
spectrum of P,_ is
Ak.,* = 2 cos(ka,*),
x__.,* = (sin(kia,,)), for k = l,... ,n, l=l,...,m.
Since a,* ranges from zero to 7r as k varies from 1 to n, the eigenvalues are ordered
The principal eigenvector z_l.,, = (sin(i0,_) ) has all positive components, as required by
Perron-Frobenius theory. A plot of the components of this vector against vertex numbers
shows a half sine curve, with its maximum near n/2, and near zero components at either ends.14
The eigenvectorcorrespondingto the second largest eigenvector is z__2,n = (sin(2i0,) ), and
a plot of this vector against vertex numbers shows a full sine curve. It attains its maximum
near n/4, its minimum near 3n/4, and has near zero values at either end, and near n/2.
Recall that the components of the Fiedler vector of the path decrease monotonically from
left to right along the path. Hence a partition of the vertices with respect to some component
of the Fiedler vector yields two parts, each forming a connected subgraph. This may not be
true of a partition with respect to the second eigenvector of the adjacency spectrum, because
of its sinusoidal behavior.
Barnes's algorithm [3] for partitioning vertices into two sets such that a small edge
separator joins the sets makes use of the first two adjacency eigenvectors. If we apply
Barnes's algorithm to partition the path Pz0 into two sets of half the vertices each, one part
consists of the first thirteen vertices, and the last two vertices. This part has two connected
components. The second part consisting of vertices 14 to 28 forms a connected subgraph.
Now we consider the adjacency spectra of the grids. Let B, C denote the adjacency
matrices of the five and nine-point rn × n grids, respectively. It can be verified that
B = A,_®I,_+I,_®A,_,
C = A,_®I,,,,+I,_®A,_+A,_®A,..
The eigenvalues of B are
#k,Z=_k.,_+)q,_, fork=l,...,n, l=l,...,m,
while those of C are
#k,t = )_k,n + )q,,_ + _k.,_,_t,,_, for k = 1,... ,n, l = 1,... ,rn.
The corresponding eigenvectors for both grids are
Y-_,t = z-_.n ® z--t,m"
Again, for both grids, the largest eigenvalue is #x,1, and the the second largest is #2.1. The
corresponding eigenvector 2.1 is plotted in Figure 3.Since a plot of the components of the second eigenvector across a row yields a full
sinusoidal curve, as in the case of the path, a partition of the vertices into two sets with
respect to some component of this vector may result in parts consisting of disconnected
subgraphs. Barnes's algorithm applied to the 15 x 30 grid yields two connected components
in one part, and a single component in the other part.
5. A spectral partitioning algorithm. In this section we describe an algorithm for
finding a vertex separator of a graph by means of its Lapacian matrix. Recall that we require
the separator to partition the graph into two parts with nearly equal numbers of vertices in
each part, and also that the size of the vertex separator be small.
The algorithm uses the second eigenvector of the Laplacian matrix to compute the
partition. We find z,,, the median value of the components of the eigenvector. Let A' be the
set of vertices whose components are less than or equal to zm, and let B' be the remaining15
FieMlcrvectorof the fivepointgrid.
FIG. 3. The second adjacency eigenvector of the five- and nine-point grids.
set of vertices. If there is a single vertex with the component corresponding to z,_, then A'
and B' differ in size by at most one. If there are several vertices with components equal to
z,,_, arbitrarily assign such vertices to A' or B' to make these sets differ in size by at most
one.
This initial partition of G gives an edge separator in the graph. Let A1 denote the
vertices in A' which are adjacent to some vertex in B', and similarly let B1 be the set of
vertices in B' which are adjacent to some vertex in A'. Let E1 be the set of edges of G with
one endpoint in A1 and the other in B1. Then E1 is an edge separator of G. Note that the
subgraph H = (A1, B1, El) is bipartite.
We require a vertex separator of G, which can be obtained from the edge separator E1 by
several methods. The simplest method is to choose the smaller of the two endpoint sets A1
and B1. Gilbert and Zmijewski [22] have computed vertex separators from edge separators in
this manner in the context of a parallel Kernighan-Lin algorithm. Leiserson and Lewis [29]
have computed vertex separators from edge separators by a heuristic algorithm. However,
there is a better way to choose a smalles_ vertex separator which can be computed from the
given edge separator E'.
The idea is to choose a set S' consisting of some vertices from both sets of endpoints A1
and B1, such that every edge in E1 is incident on at least one of the vertices in S. The set S
is a vertex separator in the graph G since the removal of these vertices causes the deletion of
all edges incident on them, and this latter set of edges contains the edge separator El. The
set S is a vertez cover (cover) of the bipartite graph H.
16
A cover of smallest cardinatity is a minimum cover. A minimum cover S of the graph H
is a smallest vertex separator of G corresponding to the edge separator El. The separator S
can be computed by finding a maximum matching in H. Details are provided later in this
section.
In general, S wiU consist of vertices from both A1 and B1. Let A, and B, denote the
vertices of S that belong to A1 and B1, respectively. Then S separates G into two subgraphs
with vertex sets A = A'\ A,, B = B'\ B,. The Spectral Partitioning Algorithm is summarized
in Figure 4.
1. Compute the eigenvector x_2 and the median value z,, of its components;
2. Partition the vertices of G into two sets:
A' = {vertices with z,, < z,_,};
B' = V \ A;If tA'I - IB'I > 1, move enough vertices with components equal to z,,
from A' to B' to make this difference at most one;
3. Let A1 be the set of vertices in A' adjacent to some vertex in B';
Let B1 be the set of vertices in B' adjacent to some vertex in A';
Compute//= (A1, B1, El), the bipartite subgraph induced by the vertex sets A1, B1;
4. Find a minimum vertex cover S of H by a maximum matching;
Let S = A, U B0, where A° C_ A1, Bo C_ B1;
S is the desired vertex separator, and separates G into subgraphs with
vertex sets A = A' \ Ao, B = B' \ B,.
FIG. 4. The spectral partitiorting algorithm
Computing a minimum cover. We now describe how a minimum cover S of the
bipartite graph H is computed by matching techniques. In the following description, call
one set of vertices of the bipartite graph 'rows' and the other set 'columns'. A matching
in a graph is a subset of its edges which do not share a common endpoint. A mazimum
matching is a matching of maximum cardinality. A vertex is matched if it is an endpoint of
an edge in a matching, and unmatched otherwise. A bipartite graph has a perfect matching
if all rows and columns are matched in a maximum matching; necessarily, the number of
rows must be equal to be the number of columns. Figure 5 shows a maximum matching
in a bipartite graph. The matched edges are indicated with thick lines. There are several
efficient algorithms to compute maximum matchings in graphs.
A path in a graph is a sequence of distinct vertices vl, v_, ..., vk such that (vi,vi+l) is
an edge for i = 1, ..., k - 1. An alternating path with respect to a matching M is a path
which has alternate edges in M; by definition of a matching, then the remaining edges do
not belong to M.
With respect to a maximum matching M in a bipartite graph //, we can define the
following sets. The reader may find Figure 5 helpful to understand these definitions. Let17
R1
R2
R3
(Z
I _ b Ci
2 c C_
3 d
C3
4
Rows Columns
FIG. 5. A mazimum ma_chin9 and a minimum cover in a bipartite graph.
R_ (Cl) denote the set of rows (columns) reachable by alternating path from all unmatched
columns. All rows in R1 are matched since M is a maximum matching. We include all
unmatched columns in the set C1, since these columns are reachable by alternating paths of
length zero. Similarly, let C3 (R3) denote the set of columns (rows) reachable by alternating
path from all unmatched rows. Again, all columns in C3 are matched, and R3 includes all
unmatched rows. There may be rows and columns that have not been visited by alternating
paths; include such columns in C2 and rows in R2. _.From the definitions of C1 and R3, it can
be shown that all vertices in R2 and C2 are matched, these sets have the same caxdinality,
and that they are perfectly matched to each other.
It should be noted that in a given bipartite graph, some of these sets may be absent. For
instance, if there are no unmatched columns in a maximum matching, then the sets C1 and
R1 will be absent. Similarly, if there are no unmatched rows, Rs and C3 will be absent. If
alternating paths from unmatched rows and unmatched columns visit all rows and columns,
the sets R2 and C2 will be empty.
Maximum matchings and minimum covers are dual concepts in bipartite graphs, and a
minimum cover S of H can be obtained as either R1 U R2 t.JCs, or R1 t3 C2 t3 6'3. These provide
two different vertex separators in the original graph G, if R2 and 6'2 are not empty, and this
freedom can be used to choose a separator that makes the parts A and B less unequal. In
the bipartite graph in Fig. 5, a minimum cover is either the set {1, 2, d} or the set {1, c, d}
with cardinality three.These row sets and column sets can be used to compute a canonical decomposition of
bipartite graphs called the Dulmage-Mendelsohn decomposition. This decomposition induces
a block triangular form of sparse matrices, which is useful in several sparse matrix algorithms.
A detailed description may be found in [39].
Complexity of the algorithm. Now we consider the complexity of each step of this
algorithm.The dominant cost is the cost of the Lanczos algorithm for computing the second eigen-
18
TABLE 1
Partition8 using median component of the second Laplacian eigenvec_or.
key
BCSPWR09
BCSPWR10
BCSSTK13
CAN 1072
DWT 2680
JAGMESH
LSHP3466
NASA1824
NASA2146
NASA4704
GRD61.101.5
GRD61.101.9
GRDS0.80.5
GRD80.S0.9
edge
separator
IEI[ IA'l IB'[
34 862 861
44 2,650 2,650
3,585 1,002 1,001
165 536 536
85 1,340 1,340
50 468 468
121 1,733 1,733
740 912 912
934 1,073 1,073
1,324 2,352 2,352
endpoint set
61 3,111 3,050
181 3,111 3,050
80 3,200 3,200
238 3,200 3,200
vertex separators
matching
[Axl IA'] - ]AI] IB'l
22 840 861
35 2,615 2,650
295 707 1,001
53 483 536
29 1,311 1,340
26 442 468
61 1,672 1,733
103 809 912
96 977 1,073
185 2,167 2,352
61 3,050 3,050
61 3,050 3,050
80 3,120 3,200
80 3,120 3,200
[SI IAI IBI
20 857 846
31 2,623 2,646
236 862 905
33 525 514
28 1,313 1,339
26 442 468
61 1,672 1,733
102 839 883
74 1,036 1,036
172 2,266 2,266
61 3,050 3,050
61 3,050 3,050
80 3,120 3,200
80 3,120 3,200
vector, which costs O(e) flops per iteration.
We find the median component of the second eigenvector of the Laplacian matrix by an
algorithm that selects the k-th element out of n. This algorithm finds the desired element
by repeatedly partitioning a subarray with respect to a pivot element, and does not sort the
array. This algorithm is O(n) in the average case. and f_(n 2) in the worst case. The selection
of the k-th element can be done in linear time in the worst case by an algorithm designed
by Blum, Floyd, Pratt, Rivest, and Tarjan [8].
The partition into the sets A and B can be done in O(n) time. The bipartite graph
H can be generated in O(e) time, by examining the adjacency list of each vertex at most
once. Let m be the smaller of [A'] and [B'[, and let e' - IE'[. A maximum matching and
the minimum cover S can be obtained in O(x/_e t) time by an algorithm of Hopcroft and
Karp [27]; this algorithm has been efficiently implemented by Duff and Wiberg [14].
6. Results. In this section, we report computational results obtained from the Spectral
Partitioning Algorithm and provide comparisons with the Nested Dissection Algorithm, the
Kernighan-Lin Algorithm, the Fiduccia-Mattheyses Algorithmas implemented by Leiserson
and Lewis [29], and the vertex partitioning based on the multiple minimum degree algorithm
by Liu [31]. Several sparse matrices from the Boeing-Harwell collection [13] and five- and
nine-point grids are partitioned using these algorithms.
The partitions obtained with the Spectral Partitioning Algorithm are tabulated in Ta-
ble 1. The edge separator E1 separates the graph into two parts A' and B _. The sizes of these
19
setsare shownin the first group of three columns in the table. We show two vertex separa-
tors obtained from El: the first vertex separator is chosen to be the the smaller endpoint set
of Et; in the table, this set is denoted A1. The second vertex separator S includes subsets
of vertices from both endpoint sets, and is computed by means of a maximum matching to
be a minimum vertex cover of the bipartite graph induced by El.
For the Boeing-Harwell problems, the matching method computes vertex separators that
are, on the average, about 11.5 percent smaller than the separators obtained from the smaller
endpoint set. Since there are two choices for the minimum cover, a good choice also makes
the two part sizes less different.
The edge separators obtained are small relative to the total number of edges in each
graph, except for the BCSSTK13 problem, which has a high average degree. For all problems,
except two, the vertex separators obtained are also relatively small (s < 0.04) in comparison
to the parts generated by the separators. The exceptions are BCSSTK13 and NASA1824.
Both these problems have large second eigenvalue _2. For BCSSTK13, _2 ,_ 0.65; in contrast,
for the 80 × 80 nine-point grid, which has good separators, 3t2 _ 4.6 × 10 -s.
For the grid graphs, vertex separators can be computed by explicitly computing the
second eigenvector by the methods in Section 4. Here, we investigate the partitions obtained
by the Spectral Partitionin 9 Algorithm with this eigenvector computed by the Lanczos algo-
rithm. We partitioned the 61 × 101 grids initially into two sets with 3050 (50 columns) and
3111 (51 columns) vertices. The edge separator obtained joins vertices in the fiftieth column
to vertices in the fifty first column. The vertex separator computed is the middle (fifty first)
column.
In the square grids, the second eigenvalue has geometric multiplicity two, and there are
two linearly independent eigenvectors. The eigenvectors in Section 4, _,1' and V_1,2,obtained
by the Kronecker products of the Laplacian eigenvectors of the path, can be used to compute
two sets of edge separators. One edge separator joins vertices in the fortieth column to
vertices in the forty first column, and the other joins vertices in the fortieth row to the forty
first row. In general, the Lanczos algorithm will compute a linear combination of the two
eigenvectors described above, leading to a different (and larger) edge separator. However,
with the starting vector we used, the Lanczos algorithm converged to the eigenvector _,2'
and the latter edge separator was computed. The computed vertex separator consists of the
vertices in the fortieth row.
We are interested in comparing the quality of the separators computed by the spectral
algorithm with the separators computed from a modified Automated Nested Dissection al-
gorithm. The nested dissection routine in SPARSPAK, FNSEP, finds a pseudo-peripheral
vertex in the graph, and generates a level structure from it. It then chooses the median
level in the level structure as the vertex separator. However, this choice may separate the
graph into widely disparate parts. We modified this routine such that the vertex separator
is chosen to be the smallest level k such that the first k levels together contain more than
half the vertices. A vertex separator is obtained by removing from the vertices in level k
those vertices which are not adjacent to any vertex in level k + 1. By the construction of the
level structure, the removed vertices are adjacent to vertices in level k - 1, and hence these
are added to the part containing vertices in the first k - 1 levels. The other part has vertices
2o
TXBLE 2
Partitions from Automated Nested Dissection.
key
BCSPWR09
BCSPWR10
BCSSTK13
CAN 1072
DWT 2680
JAGMESH
LSHP3466
NASA1824
NASA2146
NASA4704
GRD61.101.5
GRD61.101.9
GRDS0.80.5
GRD80.80.9
vertex
separator
ISl IAI [BI
68 762 893
169 2,421 2,710
302 764 937
64 478 530
28 1,327 1,325
26 455 455
59 1,711 1,696
137 839 848
131 1,008 1,007
296 2,245 2,163
61 3,050 3,050
iii 3,025 3,025
80 3,160 3,160
113 3,136 3,151
edge
separators
JEll [AI IB U S[ IE21 IA U SI IBI
80 762 961
209 2,421 2,879
3,035 764 1,239
108 478 594
84 1,327 1,353
50 455 481
118 1,711 1,755
910 839 985
1,473 1,008 1,138
2,134 2,245 2,459
121 3,050 3,111
327 3,025 3,131
158 3,160 3,240
333 3,136 3,264
130 830 893
317 2,590 2,710
4,792 1,066 937
342 542 530
84 1,355 1,325
50 481 455
116 1,770 1,696
1,347 976 848
1,569 1,139 1,007
2,424 2,541 2,163
121 3,111 3,050
333 3,131 3,025
158 3,240 3,160
339 3,249 3,151
in levels k + 1 and higher. Statistics about the vertex separators so obtained are shown in
Table 2.
We can also obtain two edge separators using the level structure from the set of edges
joining the vertex separator to the two parts A and B. The sizes of the edge separators and
the part sizes they generate are also shown in Table 2.
The Spectral Partitioning Algorithm computes smaller vertex separators than the Nested
Dissection Algorithm; on the average, the spectral vertex separators are about 65 percent of
the nested dissection vertex separators. The spectral algorithm also succeeds in keeping the
part sizes less disparate than the latter algorithm. The average difference in the part sizes
is about 7 percent for nested dissection, but there are problems for which this difference is
greater than 20 percent.
For most problems, the spectral algorithm also finds smaller edge separators in the graph
than nested dissection. There are a few problems where the best edge separator obtained
by nested dissection is smaller than that obtained by the spectral algorithm, but the former
edge separators separate the graph into parts with widely differing sizes. In the spectral
algorithm, equal part sizes can be obtained by partitioning with respect to the median
eigenvector component; any other choice of part sizes can also be obtained by partitioning
with respect to the appropriate component. Since edge separators are computed in nested
dissection by means of a level structure, part sizes cannot be controlled as effectively.
For the five-point grids, the vertex separators obtained by the Nested Dissection Algo-
rithm correspond to a 'diagonal' of the grid. For the nine point grids, the algorithm fails to
21
find the smallest vertex separators. For all the grid problems, automated nested dissection
fails to find the optimal edge separators.
6.1. Kernighan-Lin Algorithm. The Kernighan-Lin Algorithm is a heuristic algo-
rithm that can be used to compute small edge separators. We investigated the use of this
algorithm separately and in conjunction with the Spectral Partitioning Algorithm, to com-
pute edge and vertex separators.
The Kernighan-Lin algorithm begins with an initial partition of the graph into two
subsets A', B' which differ in their sizes by at most one. At each iteration, the algorithm
chooses two subsets of equal size to swap between A and B, thereby reducing the number
of edges which join A to B. We refer the reader to Kernighan and Lin [28], or Gilbert and
Zmijewski [22] for a detailed description of how the algorithm chooses the subsets to be
swapped. The algorithm terminates when it is no longer possible to decrease the size of the
edge separator by swapping subsets.
One initial partition we could use is the edge partition obtained from the Spectral Parti-
tioning Algorithm, and a second choice is to use a randomly computed initial partition. We
consider the four graphs with the largest edge separators from Table 1, and report the sizes
of the edge and vertex separators obtained with the Kernighan-Lin algorithm in Table 3.
An edge separator was computed first, and then a vertex separator was obtained as before
by matching methods. The column labeled 'SP' corresponds to the output of the spectral
algorithm, 'SP,KL' corresponds to the Kernighan-Lin algorithm with initial partition from
the spectral algorithm, and 'KL' corresponds to the Kernighan-Lin algorithm with a random
initial partition.
Application of the Kernighan-Lin algorithm with the spectral partition as input succeeds
in reducing the sizes of the edge separator considerably for two of the four problems. Thus if
one is primarily concerned with small edge separators, applying the Kernighan-Lin algorithm
to the partition produced by spectral algorithm could be worthwhile. However, the size of
the vertex separator is not improved. For two of the problems, the size remains the same;
for a third, it decreases by one, and the size increases for a fourth problem. Also, for two of
the four problems, the spectral algorithm by itself finds better vertex separators than those
obtained by the Kernighan-Lin algorithm alone.
Gilbert and Zmijewski [22] have observed that the quality of the partition found by the
Kernighan-Lin algorithm strongly depends on the quality of the initial partition. They show
for a grid graph that it is possible to choose a bad initial partition for the Kernighan-Lin
algorithm such that the algorithm will not find a minimum edge separator.
Edge separators obtained from the Kernighan-Lin algorithm with initial spectral par-
tition are better than those obtained from the application of the Kernighan-Lin algorithm
with random initial partitions for two of the four problems. Use of the initial partition from
spectral algorithm also helps the Kernighan-Lin algorithm to converge faster. On these four
problems, the Kernighan-Lin algorithm ran on the average about 3.2 times faster when the
spectral partition was used. Thus the spectral algorithm could be used to generate initial
partitions of high quality for the Kernighan-Lin algorithm.
22
TABL_ 3
Partitions from the Kernighan-Lin Algorithm. The first table describes the edge separators, and the second,
vertez separators.
key IA'I IB'I IExlSP SP,KL KL
1,002 1,001
912 912
1,073 1,073
2,352 2,352
BCSSTK13
NASAl824
NASA2146
NASA4704
3,585 2,880 3,550
740 739 739
934 870 870
1,324 1,313 1,525
key SP SP,KL KL
ISI IAI Inl ISl IAI Inl ISl IAI 1nl
BCSSTK13
NASAl824
NASA2146
NASA4704
236 862 905
103 839 883
74 1,036 1,036
172 2,266 2,266
250 870 883
102 830 892
74 1,036 1,036
172 2,266 2,266
284 772 947
102 830 892
74 1,036 1,036
204 2,163 2,337
6.2. Comparison with Leiserson-Lewis' and Liu's results. In [29] Leiserson and
Lewis have used the Fiduccia-Mattheyses Algorithm [16] to compute vertex separators and
then to order sparse matrices. Liu [31] uses the multiple minimum degree ordering algorithm
to compute vertex separators, and then improves the separator (by decreasing its size and
making the parts less unequal) by a matching technique. He uses his separator algorithm
in [30] to compute a good ordering for parallel factorization. In both implementations
sparse matrices from the Boeing-Harwell Collection are used, so we are able to give a direct
comparison of the top level vertex bisector here. The figures in Table 4 are obtained directly
from Liu's report [31] and from Lewis (personal communication). In both cases we have
added small disconnected components, which were created by the vertex separators, to the
smaller of the two sets IAI or IBI. For the Leiserson-Lewis results we list in some instances
two different partitions, where it was difficult to judge which one should be considered better.
The results in Table 4 show that the current sparse matrix algorithms yield very good
partitions, which generally can be judged to better than the ones produced by the Spectral
Partitioning Algorithm. However, neither one of the two algorithms offers any easy prospect
of a parallel implementation. Another factor which cannot be evaluated in this comparison
is the relative excecution time of the algorithms.
7. Convergence. The dominant computation in the Spectral Partitioning Algorithm is
the computation of the second eigenvector of the Laplacian matrix by the Lanczos algorithm.
Since the Lanczos algorithm is an iterative algorithm, the number of iterations and the time
23
TABLE 4
Ve_ez separators f'eora _he Leiserson-Lewis and fhe Liu algom_hms.
key Leiserson-Lewis Liu
iS[ IAI ]BI ISl [A[ IBI
BCSPWR09
BCSPWR10
BCSSTK13
CAN10724
DWT2680
LSHP3466
7 858 854
18 2641 2634
228 1001 774
242 892 869
31 536 506
34 522 516
28 1339 1313
46 2381 1139
57 1708 1701
8 1026 689
19 2661 2620
298 941 764
38 665 368
26 1369 1283
61 1727 1678
required to compute this eigenvector is dependent on the number of correct digits needed in
the eigenvector components. In this section, we describe the details of an implementation of
the Lanczos algorithm for computing this eigenvector, and study how the quality of computed
separators depend on the accuracy in the second eigenvector.
7.1. The Lanczos algorithm. The most efficient algorithm for computing a few eigen-
values and eigenvectors of large, sparse symmetric matrices is the Lanczos algorithm. Since
the Lanczos algorithm is discussed extensively in the textbook literature [23, 35], we do
not include a detailed description of the standard algorithm here. The convergence of the
Lanczos algorithm depends critically on the distribution of the eigenvalues of the underlying
matrix. Usually the extreme eigenpairs, i.e. the largest and smallest, are found first. How-
ever it is also known that for operators such as the discrete Laplacian for a grid problem,
or more generally for positive definite finite element matrices which are approximations to
elliptic operators, the Lanczos algorithms converges in most cases to the extreme right, i.e.
the very large eigenvalues, before delivering good approximations to the eigenvalues close
to zero. This behavior can be explained with the so-called Kaniel-Paige-Saad theory (see
[35]). When computing the smallest positive eigenvalue of the Laplacian matrix Q, one faces
exactly the same situation: the Lanczos algorithm delivers very good approximations to
the large eigenvalues before converging to the desired second smallest eigenvalue. Thus the
Lanczos algorithm potentially requires long runs before it computes an approximation to the
second eigenpair.
Several potential modifications can be incorporated in the simple Lanczos algorithm
for the faster computation of the second eigenvector, but unfortunately none of them is
applicable here. The first obvious improvement of the situation would be to apply the
shifted and inverted operator, i.e. to consider the eigenvalue problem
(8) (Q - _I)-1u = _ u.
This is a standard technique in finite element applications [24], and has been used very
successfully in a variety of implementations of the Lanczos algorithm [15, 25, 36, 42]. The24
eigenvectors remain the same, and the eigenvalues # of (8) are related to the eigenvalues A of
Q by a simple transformation. In the situation here, a shift o" chosen near zero, would result
in a very rapid convergence to the eigenvalue )_2. This approach cannot be taken here, since
it requires the factorization of the matrix Q - _I, which is a large sparse symmetric matrix
with the same sparsity structure as M. Our original goal, however, is to find an efficient
reordering of M, so to be able to factor it efficiently. Hence the 'shift and invert' approach
would require us to factor a matrix closely related to M, and thus cannot be considered in
this application.
Reorthogonalization methods are a second set of techniques, which have been applied to
the Lanczos algorithm, in order to improve both its reliability and computational efficiency
[37, 43, 36]. For the computation of the second eigenvalue and vector, only a limited amount
of reorthogonalization is necessary. No reorthogonalizations are performed at the right end of
the spectrum, with respect to the large eigenvalues, since there is no interest in the accurate
computation of eigenvalues at this end. Also it is unlikely that preserving orthogonality at
the right end will have any impact on the convergence of the Lanczos algorithm towards the
second smallest eigenvalue which is at the left end of the spectrum.
However, the first eigenvector z_t of Q is e__,the vector of all ones, and this vector can
be used for reorthogonalization at the left end of the spectrum. At each step we explicitly
orthogonalize the current Lanczos vector against _e. This is effectively a deflation of the
problem and now the eigenpair Az,z__ will be computed as the first eigenpair on the left end
of the spectrum.
Another important consideration for the Lanczos algorithm is the choice of a starting
vector. In the absence of any other information, a random starting vector is appropriate.
However, many practical matrix problems are presented already in an ordering relevant to
the formulation of the problem, sometimes even in an ordering which is close to a good band
or envelope ordering. In this case it is desirable to transmit this ordering information to
the Lanczos algorithm. This was accomplished by setting the starting vector in the Lanczos
algorithm to r__,with rl = i - (rt + 1)/2. In this way the starting vector looked very much
like a Fiedler vector for the original ordering of the matrix as presented to the algorithm. In
most cases this turned out to be a better choice than a random starting vector, because it
resulted in faster convergence to the second eigenvector.
Finally, another point needs to mentioned. Considering the very simple structure of the
Laplacian matrix Q, and the seeming simplicity of the task of computing just one eigenpair
at the left end of the spectrum, one might be inclined to avoid the complexities of the Lanczos
algorithm, and attempt to solve this problem with a simple shifted power method, and a
deflation procedure analogous to the one described above. This was tried as a first attempt
at the computation of a second eigenvector, but with very poor results. The power method
converged exceedingly slowly, in many cases exhibiting the phenomenon of rnisconvergence
[38]. This meant that the power method settled down at an eigenvalue of Q, which was not
the Fiedler value, and whose eigenvector correspondingly delivered a very poor reordering.
The results here prove the claims of [38] to be correct that even in the simplest cases the
Lanczos algorithm is the method of choice, when computing eigenvalues of large, sparse,
symmetric matrices.
25
The above discussion can now be summarized in the following more complete description
of the special form of the Lanczos algorithm:
ALGORITHM 1.
1. Given the sparsity structure of a matriz M, form the Laplacian matriz Q.
2. Pick a starting vector 7., with ri = i -- (n + 1)/2.
3. Carry out a Lanczos iteration with the matriz Q and starting vector r__. At each
step orthogonalize the Lanczos vector against the vector e_. Stop when a second
eigenvector has been determined to sufficient accuracy.
In the algorithm described above, we have assumed that the Laplacian Q(G) is irre-
ducible, or equivalently that the graph G is connected. Many of the sparse matrices from
the Boeing-Harwell collection have disconnected adjacency graphs. We can apply the above
algorithm to each connected component separately, but there is a better approach, which we
now describe.
If a graph G has k connected components, then its Laplacian Q(G) has its first k
eigenvalues A1, A2, ..., /Xk equal to zero. For each connected component Cj of G, let z_j
be the vector with ones in components corresponding to the vertices in Cj, and zeros in all
other components. The vectors z__l, z__, ..., z_a, can be chosen to be independent eigenvectors
corresponding to the multiple eigenvalue zero. The next smallest eigenvalue Ak+l is positive,
and a corresponding eigenvector z_k+l is the vector used to partition the graph.
At each step of the Lanczos algorithm, we can now orthogonalize the Lanczos vector
simultaneously against the first k eigenvectors z__l, ..., z_h, since these vectors have nonzeros
in disjoint rows. This approach has the advantage that it increases the vector length in the
Lanczos algorithm, and thus permits efficient vectorization.
7.9.. Convergence and quality of separators. We now present our results on the
number of iterations and the time required by the Lanczos algorithm as the second eigen-
vector is computed to a set of different tolerances. The tolerance criterion, tol, is the 2-norm
of the residual vector Qu_ - Au_, where ,X, u_ are the computed quantities at the current step
in the algorithm. We also study the quality of the vertex separators obtained from these
approximate eigenvectors.
We report results for a few representative problems from the Boeing-HarweU collection
and for two grid problems in Table 5. The number of iterations reported are multiples of
twelve, since we checked for convergence in "the Lanczos algorithm by an eigendecomposition
of the tridiagonal matrix only after every twelve iterations. Times are in seconds on a Cray
Y-MP, using our vectorized Lanczos code. For each value of tol, we report the size of the
vertex separator and the corresponding part sizes computed by the Spectral Partitioning
Algorithm. Blank entries in the separator columns mean that the separator computed is the
same as the one obtained with the previous tolerance.
For most of the problems that we have computational results, it is only necessary to
compute the second eigenvector to a tolerance of about 10 -2, to obtain the best separator
obtained by the spectral algorithm. This accuracy only requires a modest number of Lanczos
iterations, and can be obtained reasonably fast. One class of notable exceptions is the power
network problems, illustrated by BCSPWR10 in the table. In these problems, a vertex
26
TABLE5Convergence Resul_8. Times are in seconds on a C'ray Y-MP.
Key
NASA4704
BCSSTK13
BCSPWR10
Wol Items Time
10 -1 0.27
10-2
10-3
10-4
10-5
10-6
10-1
10-2
10-3
10 -4
10-5
10-8
10-I
10-2
10-3
10 -4
GRD61.101.5 10 -2
10-3
10-4
10-5
10-e
GRD61.101.9 10 -1
10-2
10 -3
10 -4
10-5
10-6
24
60 0.65
72 0.80
96 1.10
108 1.30
120 1.50
36 0.23 236
36 0.23
48 0.30
60 0.39
72 0.49
84 0.60
24 0.24
84 0.92
252 7.20
300 11.90
12 0.15
36 0.42
96 1.26
108 1.47
120 1.67
12 0.16
24 0.30
108 1.53
120 1.76
144 2.38
156 2.70
ISI IAI IBI172 2266 2266
905 862
171 2619 2510
72 2642 2586
34 2643 2623
31 2646 2623
101 3050 3010
61 3050 3050
101 3030 3030
61 3050 3050
27
has a small average degree, and the graphs have large diameters, and hence computing
global information in such graphs is relatively slow. Hence a large number of iterations are
necessary to compute the second eigenvector accurately. In the BCSPWR10 problem, after
300 iterations, the norm of the residual in the eigenvalue equation was about 10 -4. In this
problem, the vertex separator decreases in size as the eigenvector becomes more accurate.
8. Conclusions. We have considered an algebraic approach to computing vertex sep-
arators, and have shown that the eigenvalues of the Laplacian matrix can be used to obtain
lower bounds on the sizes of the separators. We have described a heuristic algorithm for
computing vertex separators from the second eigenvector of the Laplacian. It is enough to
compute the eigenvector to low accuracy to obtain good separators for most problems.
Other approaches to computing vertex separators in sparse matrix algorithms have been
considered by several authors. Automatic Nested Dissection (AND) is used in SPARSPAK to
compute fill-reducing orderings of sparse matrices [20]. More recently, a parallel Kernigha_-
Lin algorithm was employed to compute orderings appropriate for parallel factorization by
Gilbert and Zmijewski [22]. The separators computed by the spectral algorithm compare
favorably with those obtained by AND or the Kernighan-Lin algorithm. The results with
the Spectral Partitioning Algorithrr_re not quite as good as those obtained by Leiserson and
Lewis [20] and Liu [31].
The spectral algorithm has an advantage over all these algorithms in that its domi-
nant computation is an eigenvector computation, and is fairly straight forward to compute
e_ciently on medium size multiprocessors used in scientific computing. For these other al-
gorithms, it is either not clear how to implement them in parallel or the degree of parallelism
is not high.
The computation of good separators is useful in any algorithm that employs the divide
and conquer paradigm. In particular the Spectral Partitioning Algorithrrtwi11 be also useful
for VLSI layout problems [5]. But our immediate intent was to use the spectral separator
algorithm to compute good orderings for parade1 sparse factorizations. In order to accomplish
our goals in this respect more work remains to be done. We intend to compute and study
the quality of orderings obtained by the recursive application of the Spectral Partitioning
Algorithm. A faster computation of the Fiedler vector using Davidson's method might be
possible. Finally much remains to be understood about the theoretical underpinneings of the
Spectral Partitioning Algorithm. How do Laplacian eigenvectors partition graphs? It will be
useful to obtain results on the quality of the partitions computed by Laplacian eigenvector
components. It will also be helpful to identify classes of graphs that are partitioned optimally
by spectral algorithms.
Even though much more needs to be done, in order to demonstrate the usefulness of the
Spectral Partitioning Algorithmfor reordering sparse matrices in parallel our results are very
encouraging for the following reasons: These results are very encouraging for the following
reasons:
1. This a completely new type of algorithm for the graph partitioning problem and
hence potentially for reordering sparse matrices.
2. Since the algorithm involves mainly floating point computation it becomes very
competitive on very large problems on machines like the Cray Y-MP, where floating
28
point arithmetic is considerably faster than integer arithmetic.
3. Since the algorithm involves mainly dense and sparse vector operations, it can be
easily implemented in parallel.
REFERENCES
[1] N. ALON, Z. GALIL, AND V. D. MILMAN, Better ezpanders and $1tperconcentrators, J. Algorithms, 8
(1987), pp. 337-347.
[2] N. ALON AND V. D. MILMAN, A1, isoperimetric inequalities for graphs, and superconcentrators, J.Comb. Theory, Series B, 38 (1985), pp. 73-88.
[3] E. R. BARNES, An algorithm/or partitioning the nodes of a graph, SIAM J. Alg. Disc. Meth., 3 (1982),
pp. 541-550.
[4] E. R. BARNES AND A. J. HOFFMAN, Partitioning, spectra, and linear programming, in Progress inCombinatorial Optimization, W. E. PuUeyblank, ed., Academic Press, 1984, pp. 13-25.
[5] S. N. BItATT AND F. T. LEIGHTON, A framework for solving VLSI graph layout problems, I. of Comp.System Sciences, 28 (1984), pp. 300 - 343.
[6] F. BIEN, Constructions of telephone networks by group representations, Notices of the Amer. Math.
Sue., 36 (1989).[7] N. L. BIGGS, Algebraic Graph Theory, Cambridge University Press, Cambridge, 1974.
[8] M. BLUM, R. W. FLOYD, V. R. PRATT, R. L. RIVEST, AND R. E. TARJAN, Time bounds for
selection,J.Comput. & Systems Sci.,7 (1972),pp. 448-461.
[9] R. B. BOPPANA, Eigenvalues and graph bisection: an average case analysis, in 28th Annual Syrup.Found. Cutup. Sci, 1987, pp. 280-285.
[10] P. BUS_.R, On the bipartition of graphs, Disc. Appl. Math., 9 (1984), pp. 105-109.[11] D. M. CVETKOVlC, M. DOOE, AND H. SACHS, Spectra of Graphs, Academic Press, 1980.
[12] W. E. DONATa AND A. J. HOFFMAN, Lower bounds for the partitioning of graphs, IBM Journal of
Research and Development, 17 (1973), pp. 420-425.
[13] I. S. DUFF, R. G. GRIMES, AND J. G. LEWIS, Sparse matriz test problems, ACM TOMS, 15 (1989),
pp. 1 - 14.
[14] I. S. DUFF AND T. WIBERG, Implementations of o(n/2r) assignment algorithms, ACM Trans. on
Math. Software, 4 (1988), pp. 267-287.[15] T. ERICSSON AND A. RUHE, The spectral transformation Lanczos method, Math. Cutup., 34 (1980),
pp. 1251-1268.
[16] C. FIDUCCIA AND R. MATTH_.YSES, A linear time heuristic for improving network partitions, in Design
Automation Conference, 1982.
[17] M. FIEDLER, Algebraic connectivity of graphs, Czech. Math. J., 23 (1973), pp. 298-305.
[18] _, A property of eigenvectors of non-negative symmetric matrices and its application to graph
theory, Czech. Math. J., 25 (1975), pp. 619-633.
[19] --, Special matrices and their applications in numerical mathematics, Martinus Nijhoff Publishers,
Dodtecht, The Netherlands, 1986.[20] J. A. GEORGE AND J. W. LIU, Computer Solution of Large Sparse Positive Definite Systems, Prentice
Hall, 1981.
[21] J. A. GEORGE AND E. NG, On row and column orderings for sparse least squares problems, SIAM J.
Numer. Anal., 20 (1983), pp. 326-344.[22] J. R. GILBERT AND E. ZMXJEWSKI, A parallel graph partitioning algorithm for a message passing
multiprocessor, in Proceedings of the Intern. Conf. Supercomput., Springer Verlag, 1987.
[23] G. H. GOLUB AND C. F. VAN LOAN, Matriz Computations, Johns Hopkins University Press, Balti-
more, MD, 1983.
[24] R. GRIMES, J. LEWIS, AND H. SIMON, Eigenvalue problems and algorithms in structural engineering,
in Large Scale Eigenvalue Problems, J. Cullum and R. Willoughby, ads., North-Holland, i986,
pp. 81-93.
[25] _, The Implementation of block Lanczos algorithm with reorthogonalization me,hods, Tech.
Rap. ETA-TR-91, Boeing Computer Services, Seattle, WA, 1988.
29
[26]A. J. HOFFMAN AND H. W. WIELANDT, The variation Of the spectrum of a normal matriz, DukeMath. J., 20 (1953), pp. 37-39.
[27] J. E. HOPCROFT AND R. M. KARP, An n 2"s algorithm for mazimum matchings in bipartite graphs,SIAM J. Comput., 2 (1973), pp. 225-231.
[28] B. W. KERNIGHAN AND S. LIN, An efficient heuristic procedure for partitioning graphs, The Bell
System Technical J., 49 (1970), pp. 291-307.
[29] C. E. LEISERSON AND J. G. LEWIS, Orderings for parallel sparse symmetric factorization, 1987, ThirdSIAM Conference on Parallel Processing for Scientific Computing.
[30] J. W. LIU, The minimum degree ordering with constraints, Tech. Rep. CS-88-02, Computer Science,York University, North York, Ontario, Canada, 1988.
[31] _, A graph partitioning algorithm by node separators, 1989, (To appear in ACM Trans. Math.
Softw.).
[32] B. MOHAR, Eigenvalues, diameter, and mean distance in graphs, April 1988, Preprint Series Dept.Math. no. 259, University E. K. of Ljubljana, Jadranska 19, 61111 Ljubljana, Yugoslavia.
[33] _, Isoperimetric numbers of graphs, 1988, (To appear in J. Comb. Theory, Series B).
[34] B. MOHAR, The laplacian spectrum of graphs, in Sixth Intern. Conf. on Theory and Appfic. of Graphs,
Kalamazoo, Michigan, 1988.
[35] B. PARLETT, The Symmetric Eigenvalue Problem, Prentice Hall, Englewood Cliffs, New Jersey, 1980.
[36] B. PARLETT, B. NOuR-OMID, AND Z. LIU, How to Maintain Semi-orthogonality Among Lanczos Vec-tors, Tech. Rep. PAM-420, Center for Pure and Applied Math., University of California, Berkeley,
CA, 1988.
[37] B. PARLETT AND D. SCOTT, The Lanczos algorithm with selective orthogonalization, Math. Comp.,
33 (1979), pp. 217-238.
[38] B. PARLETT, H. SIMON, AND L. STRINGER, Estimating the largest eigenvalue with the Lanczos algo-rithm, Math. Comp., 38 (1982), pp. 153-165.
[39] A. POTHEN AND C. FAN, Computing the block triangularforra of a sparse matriz, Tech. Rep. CS-88-51,Computer Science, Penn State, Dec 1988. To appear in ACM Trans. Math. Softw.
[40] D. POWERS, Structure of a matriz according to its second eigenvector, in Current Trends in Matrix
Theory, F. Uhlig and R. Grone, eds., Elsevier, 1987, pp. 261-266.
[41] --, Graph partitioning by eigenvectors, Lin. A.lg. Appl., 101 (1988), pp. 121-133.
[42] D. SCOTT, Block Lanczos Software for Symmetric Eigenvalue Problems, Tech. Rep. ORNL/CSD 48,
Oak Ridge National Lab, Oak Ridge, TN, 1979.
[43] H. SIMON, The Lanczos algorithm with partial reorthogonaliza_ion, Math. Comp., 42 (1984), pp. 115-136.
[44] J. WILKINSON, Modern error analysis, SIAM Rev., 13 (1971), pp. 548-fi68.
3O