Nonlinear Dimensionality Reduction by SemidefiniteProgramming and Kernel Matrix Factorization

Kilian Q. Weinberger, Benjamin D. Packer, and Lawrence K. Saul∗

Department of Computer and Information ScienceUniversity of Pennsylvania, Philadelphia, PA 19104-6389

{kilianw,lsaul,bpacker}@seas.upenn.edu

Abstract

We describe an algorithm for nonlinear di-mensionality reduction based on semidefiniteprogramming and kernel matrix factoriza-tion. The algorithm learns a kernel matrixfor high dimensional data that lies on or neara low dimensional manifold. In earlier work,the kernel matrix was learned by maximizingthe variance in feature space while preserv-ing the distances and angles between near-est neighbors. In this paper, adapting re-cent ideas from semi-supervised learning ongraphs, we show that the full kernel matrixcan be very well approximated by a productof smaller matrices. Representing the ker-nel matrix in this way, we can reformulatethe semidefinite program in terms of a muchsmaller submatrix of inner products betweenrandomly chosen landmarks. The new frame-work leads to order-of-magnitude reductionsin computation time and makes it possibleto study much larger problems in manifoldlearning.

1 Introduction

A large family of graph-based algorithms has recentlyemerged for analyzing high dimensional data thatlies or or near a low dimensional manifold [2, 5, 8,13, 19, 21, 25]. These algorithms derive low dimen-sional embeddings from the top or bottom eigenvec-tors of specially constructed matrices. Either directlyor indirectly, these matrices can be related to ker-nel matrices of inner products in a nonlinear featurespace [9, 15, 22, 23]. These algorithms can thus beviewed as kernel methods with feature spaces that “un-fold” the manifold from which the data was sampled.

∗This work was supported by NSF Award 0238323.

In recent work [21, 22], we introduced SemidefiniteEmbedding (SDE), an algorithm for manifold learningbased on semidefinite programming [20]. SDE learnsa kernel matrix by maximizing the variance in fea-ture space while preserving the distances and anglesbetween nearest neighbors. It has several interestingproperties: the main optimization is convex and guar-anteed to preserve certain aspects of the local geome-try; the method always yields a semipositive definitekernel matrix; the eigenspectrum of the kernel matrixprovides an estimate of the underlying manifold’s di-mensionality; also, the method does not rely on esti-mating geodesic distances between faraway points onthe manifold. This particular combination of advan-tages appears unique to SDE.

The main disadvantage of SDE, relative to other al-gorithms for manifold learning, is the time requiredto solve large problems in semidefinite programming.Earlier work in SDE was limited to data sets withn≈2000 examples, and problems of that size typicallyrequired several hours of computation on a mid-rangedesktop computer.

In this paper, we describe a new framework that hasallowed us to reproduce our original results in a smallfraction of this time, as well as to study much largerproblems in manifold learning. We start by showingthat for well-sampled manifolds, the entire kernel ma-trix can be very accurately reconstructed from a muchsmaller submatrix of inner products between randomlychosen landmarks. In particular, letting K denote thefull n×n kernel matrix, we can write:

K ≈ QLQT , (1)

where L is the m×m submatrix of inner products be-tween landmarks (with m�n) and Q is an n×m lineartransformation derived from solving a sparse set of lin-ear equations. The factorization in eq. (1) enables usto reformulate the semidefinite program in terms of themuch smaller matrix L, yielding order-of-magnitudereductions in computation time.

The framework in this paper has several interestingconnections to previous work in manifold learningand kernel methods. Landmark methods were orig-inally developed to accelerate the multidimensionalscaling procedure in Isomap [7]; they were subse-quently applied to the fast embedding of sparse sim-ilarity graphs [11]. Intuitively, the methods in thesepapers are based on the idea of triangulation—that is,locating points in a low dimensional space based ontheir distances to a small set of landmarks. This ideacan also be viewed as an application of the Nystrommethod [24, 12], which is a particular way of extrapo-lating a full kernel matrix from one of its sub-blocks.It is worth emphasizing that the use of landmarks inthis paper is not based on this same intuition. SDEdoes not directly estimate geodesic distances betweenfaraway inputs on the manifold, as in Isomap. As op-posed to the Nystrom method, our approach is betterdescribed as an adaptation of recent ideas for semi-supervised learning on graphs [1, 16, 18, 26, 27]. Ourapproach is somewhat novel in that we use these ideasnot for transductive inference, but for computationalsavings in a purely unsupervised setting. To managethe many constraints that appear in our semidefiniteprogramming problems, we have also adapted certainideas from the large-scale training of support vectormachines [6].

The paper is organized as follows. In section 2, we re-view our earlier work on manifold learning by semidefi-nite programming. In section 3, we investigate the ker-nel matrix factorization in eq. (1), deriving the lineartransformation that reconstructs other examples fromlandmarks, and showing how it simplifies the semidef-inite program for manifold learning. Section 4 givesexperimental results on data sets of images and text.Finally, we conclude in section 5.

2 Semidefinite Embedding

We briefly review the algorithm for SDE; more de-tails are given in previous work [21, 22]. As in-put, the algorithm takes high dimensional vectors{~x1, ~x2, . . . , ~xn}; as output, it produces low dimen-sional vectors {~y1, ~y2, . . . , ~yn}. The inputs ~xi ∈ RD areassumed to lie on or near a manifold that can be em-bedded in d dimensions, where typically d � D. Thegoal of the algorithm is to estimate the dimensional-ity d and to output a faithful embedding that revealsthe structure of the manifold.

The main idea behind SDE has been aptly describedas “maximum variance unfolding” [17]. The algorithmattempts to maximize the variance of its embedding,subject to the constraint that distances and anglesbetween nearby inputs are preserved. The resulting

transformation from inputs to outputs thus looks lo-cally like a rotation plus translation—that is, it rep-resents an isometry. To picture such a transformationfrom D=3 to d=2 dimensions, one can imagine a flagbeing unfurled by pulling on its four corners.

The first step of the algorithm is to compute the k-nearest neighbors of each input. A neighborhood-indicator matrix is defined as ηij =1 if and only if theinputs ~xi and ~xj are k-nearest neighbors or if there ex-ists another input of which both are k-nearest neigh-bors; otherwise ηij = 0. The constraints to preservedistances and angles between k-nearest neighbors canthen be written as:

||~yi − ~yj ||2 = ||~xi − ~xj ||2 , (2)

for all (i, j) such that ηij =1. To eliminate a transla-tional degree of freedom in the embedding, the outputsare also constrained to be centered on the origin:∑

i

~yi = ~0. (3)

Finally, the algorithm attempts to “unfold” the inputsby maximizing the variance

var(~y) =∑

i

||~yi||2 (4)

while preserving local distances and angles, as ineq. (2). Maximizing the variance of the embeddingturns out to be a useful surrogate for minimizing its di-mensionality (which is computationally less tractable).

The above optimization can be formulated as aninstance of semidefinite programming [20]. LetKij = ~yi · ~yj denote the Gram (or kernel) matrix of theoutputs. As shown in earlier work [21, 22], eqs. (2–4)can be written entirely in terms of the elements of thismatrix. We can then learn the kernel matrix K bysolving the following semidefinite program.

Maximize trace(K) subject to:1) K � 0.2) ΣijKij = 0.3) For all (i, j) such that ηij =1,

Kii − 2Kij + Kjj = ||~xi − ~xj ||2.

As in kernel PCA [15], the embedding is derived fromthe eigenvalues and eigenvectors of the kernel matrix;in particular, the algorithm outputs yαi =

√λαuαi,

where λα and uα are the top d eigenvalues and eigen-vectors. The dimensionality of the embedding, d,is suggested by the number of appreciably non-zeroeigenvalues.

In sum, the algorithm has three steps: (i) computingk-nearest neighbors; (ii) computing the kernel matrix;

and (iii) computing its top eigenvectors. The compu-tation time is typically dominated by the semidefiniteprogram to learn the kernel matrix. In earlier work,this step limited us to problems with n≈ 2000 exam-ples and k≤ 5 nearest neighbors; moreover, problemsof this size typically required several hours of compu-tation on a mid-range desktop computer.

3 Kernel Matrix Factorization

In practice, SDE scales poorly to large data setsbecause it must solve a semidefinite program overn × n matrices, where n is the number of examples.(Note that the computation time is prohibitive de-spite polynomial-time guarantees1 of convergence forsemidefinite programming.) In this section, we showthat for well-sampled manifolds, the kernel matrixK can be approximately factored as the product ofsmaller matrices. We then use this representation toderive much simpler semidefinite programs for the op-timization in the previous section.

3.1 Sketch of algorithm

We begin by sketching the basic argument behind thefactorization in eq. (1). The argument has three steps.First, we derive a linear transformation for approxi-mately reconstructing the entire data set of high di-mensional inputs {~xi}n

i=1 from m randomly chosen in-puts designated as landmarks. In particular, denotingthese landmarks by {~µα}m

α=1, the reconstructed inputs{xi}n

i=1 are given by the linear transformation:

xi =∑α

Qiα~µα. (5)

The linear transformation Q is derived from a sparseweighted graph in which each node represents an inputand the weights are used to propagate the positions ofthe m landmarks to the remaining n−m nodes. Thesituation is analogous to semi-supervised learning onlarge graphs [1, 16, 18, 26, 27], where nodes representlabeled or unlabeled examples and transductive infer-ences are made by diffusion through the graph. In oursetting, the landmarks correspond to labeled exam-ples, the reconstructed inputs to unlabeled examples,and the vectors ~µα to the actual labels.

Next, we show that the same linear transformationcan be used to reconstruct the unfolded data set—thatis, after the mapping from inputs {~xi}n

i=1 to outputs

1For the examples in this paper, we used the SDP solverCSDP v4.9 [4] with time complexity of O(n3+c3) per it-eration for sparse problems with n×n target matrices andc constraints. It seems, however, that large constant factorscan also be associated with these complexity estimates.

{~yi}ni=1. In particular, denoting the unfolded land-

marks by {~α}mα=1 and the reconstructed outputs by

{yi}ni=1, we argue that ~yi ≈ yi, where:

yi =∑α

Qiα~α. (6)

The connection between eqs. (5–6) will follow fromthe particular construction of the weighted graph thatyields the linear transformation Q. This weightedgraph is derived by appealing to the symmetries oflinear reconstruction coefficients; it is based on a simi-lar intuition as the algorithm for manifold learning bylocally linear embedding (LLE) [13, 14].

Finally, the kernel matrix factorization in eq. (1) fol-lows if we make the approximation

Kij = ~yi · ~yj ≈ yi · yj . (7)

In particular, substituting eq. (6) into eq. (7) givesthe approximate factorization K ≈ QLQT , whereLαβ = ~

α · ~β is the submatrix of inner products be-tween (unfolded) landmark positions.

3.2 Reconstructing from landmarks

To derive the linear transformation Q in eqs. (5–6),we assume the high dimensional inputs {~xi}n

i=1 arewell sampled from a low dimensional manifold. In theneighborhood of any point, this manifold can be locallyapproximated by a linear subspace. Thus, to a goodapproximation, we can hope to reconstruct each inputby a weighted sum of its r-nearest neighbors for somesmall r. (The value of r is analogous but not necessar-ily equal to the value of k used to define neighborhoodsin the previous section.) Reconstruction weights canbe found by minimizing the error function:

E(W ) =∑

i

∣∣∣∣∣∣~xi −∑

jWij~xj

∣∣∣∣∣∣2 , (8)

subject to the constraint that∑

j Wij = 1 for all j, andwhere Wij = 0 if ~xj is not an r-nearest neighbor of ~xi.The sum constraint on the rows of W ensures that thereconstruction weights are invariant to the choice ofthe origin in the input space. A small regularizer forweight decay can also be added to this error functionif it does not already have a unique global minimum.

Without loss of generality, we now identify the first minputs {~x1, ~x2, . . . , ~xm} as landmarks {~µ1, ~µ2, . . . , ~µm}and ask the following question: is it possible to recon-struct (at least approximately) the remaining inputsgiven just the landmarks ~µα and the weights Wij? Forsufficiently large m, a unique reconstruction can be ob-tained by minimizing eq. (8) with respect to {~xi}i>m.

To this end, we rewrite the reconstruction error as afunction of the inputs, in the form:

E(X) =∑ij

Φij ~xi ·~xj , (9)

where Φ = (In−W )T (In−W ) and In is the n × nidentity matrix. It is useful to partition the matrix Φinto blocks distinguishing the m landmarks from theother (unknown) inputs:

m︷ ︸︸ ︷ n−m︷ ︸︸ ︷Φ =

(Φ``

Φu`Φ`u

Φuu

)(10)

In terms of this matrix, the solution with minimumreconstruction error is given by the linear transforma-tion in eq. (5), where:

Q =(

Im

(Φuu)−1Φul

). (11)

An example of this minimum error reconstruction isshown in Fig. 1. The first two panels show n=10000inputs sampled from a Swiss roll and their approxi-mate reconstructions from eq. (5) and eq. (11) usingr=12 nearest neighbors and m=40 landmarks.

Intuitively, we can imagine the matrix Φij in eq. (9)as defining a sparse weighted graph connecting nearbyinputs. The linear transformation reconstructing in-puts from landmarks is then analogous to the mannerin which many semi-supervised algorithms on graphspropagate information from labeled to unlabeled ex-amples.

To justify eq. (6), we now imagine that the data sethas been unfolded in a way that preserves distancesand angles between nearby inputs. As noted in previ-ous work [13, 14], the weights Wij that minimize thereconstruction error in eq. (8) are invariant to trans-lations and rotations of each input and its r-nearestneighbors. Thus, roughly speaking, if the unfoldinglooks locally like a rotation plus translation, then thesame weights Wij that reconstruct the inputs ~xi fromtheir neighbors should also reconstruct the outputs ~yi

from theirs. This line of reasoning yields eq. (6). Italso suggests that if we could somehow learn to faith-fully embed just the landmarks in a lower dimensionalspace, the remainder of the inputs could be unfoldedby a simple matrix multiplication.

3.3 Embedding the landmarks

It is straightforward to reformulate the semidefiniteprogram (SDP) for the kernel matrix Kij = ~yi · ~yj

in section 2 in terms of the smaller matrixLαβ = ~

α · ~β . In particular, appealing to the factor-ization K ≈ QLQT , we consider the following SDP:

Maximize trace(QLQT ) subject to:1) L � 0.2) Σij(QLQT )ij = 0.3) For all (i, j) such that ηij =1,

(QLQT )ii−2(QLQT )ij +(QLQT )jj ≤ ||~xi − ~xj ||2.

This optimization is nearly but not quite identical tothe previous SDP up to the substitution K ≈ QLQT .The only difference is that we have changed the equal-ity constraints in eq. (2) to inequalities. The SDP insection 2 is guaranteed to be feasible since all the con-straints are satisfied by taking Kij = ~xi ·~xj (assumingthe inputs are centered on the origin). Because thematrix factorization in eq. (1) is only approximate,however, here we must relax the distance constraintsto preserve feasibility. Changing the equalities to in-equalities is the simplest possible relaxation; the trivialsolution Lαβ =0 then provides a guarantee of feasibil-ity. In practice, this relaxation does not appear tochange the solutions of the SDP in a significant way;the variance maximization inherent to the objectivefunction tends to saturate the pairwise distance con-straints, even if they are not enforced as strict equali-ties.

To summarize, the overall procedure for unfolding theinputs ~xi based on the kernel matrix factorizationin eq. (1) is as follows: (i) compute reconstructionweights Wij that minimize the error function in eq. (8);(ii) choose landmarks and compute the linear trans-formation Q in eq. (11); (iii) solve the SDP for thelandmark kernel matrix L; (iv) derive a low dimen-sional embedding for the landmarks ~

α from the eigen-vectors and eigenvalues of L; and (v) reconstruct theoutputs ~yi from eq. (6). The free parameters of the al-gorithm are the number of nearest neighbors r usedto derive locally linear reconstructions, the numberof nearest neighbors k used to generate distance con-straints in the SDP, and the number of landmarks m(which also constrains the rank of the kernel matrix).In what follows, we will refer to this algorithm as land-mark SDE, or simply `SDE.

`SDE can be much faster than SDE because its mainoptimization is performed over m×m matrices, wherem � n. The computation time in semidefinite pro-gramming, however, depends not only on the matrixsize, but also on the number of constraints. An ap-parent difficulty is that SDE and `SDE have the samenumber of constraints; moreover, the constraints inthe latter are not sparse, so that a naive implementa-tion of `SDE can actually be much slower than SDE.This difficulty is surmounted in practice by solving thesemidefinite program for `SDE while only explicitlymonitoring a small fraction of the original constraints.To start, we feed an initial subset of constraints to

Figure 1: (1) n = 10000 inputs sampled from aSwiss roll; (2) linear reconstruction from r = 12 near-est neighbors and m = 40 landmarks (denoted byblack x’s); (3) embedding from `SDE, with distanceand angle constraints to k=4 nearest neighbors, com-puted in 16 minutes.

the SDP solver, consisting only of the semidefinite-ness constraint, the centering constraint, and the dis-tance constraints between landmarks and their near-est neighbors. If a solution is then found that vio-lates some of the unmonitored constraints, these areadded to the problem, which is solved again. Theprocess is repeated until all the constraints are sat-isfied. Note that this incremental scheme is made pos-sible by the relaxation of the distance constraints fromequalities to inequalities. As in the large-scale train-ing of support vector machines [6], it seems that manyof the constraints in `SDE are redundant, and simpleheuristics to prune these constraints can yield order-of-magnitude speedups. (Note, however, that the cen-tering and semidefiniteness constraints in `SDE are al-ways enforced.)

4 Experimental Results

Experiments were performed in MATLAB to evaluatethe performance of `SDE on various data sets. TheSDPs were solved with the CSDP (v4.9) optimizationtoolbox [4]. Of particular concern was the speed andaccuracy of `SDE relative to earlier implementationsof SDE.

The first data set, shown in the top left panel of Fig. 1,consisted of n=10000 inputs sampled from a three di-mensional “Swiss roll”. The other panels of Fig. 1show the input reconstruction from m=40 landmarksand r = 12 nearest neighbors, as well as the embed-dingobtained in `SDE by constraining distances andangles to k = 4 nearest neighbors. The computationtook 16 minutes on a mid-range desktop computer.Table 2 shows that only 1205 out of 43182 constraints

word four nearest neighbors

one two, three, four, sixmay won’t, cannot, would, willmen passengers, soldiers, officers, lawmakersiraq states, israel, china, noriegadrugs computers,missiles, equipment, programsjanuary july, october, august, marchgermany canada, africa, arabia, marksrecession environment, yen, season, afternooncalifornia minnesota, arizona, florida, georgiarepublican democratic, strong, conservative, phonegovernment pentagon, airline, army, bush

Table 1: Selected words and their four nearest neigh-bors (in order of increasing distance) after nonlineardimensionality reduction by `SDE. The d = 5 dimen-sional embedding of D = 60000 dimensional bigramdistributions was computed by `SDE in 35 minutes(with n=2000, k=4, r=12, and m=30).

had to be explicitly enforced by the SDP solver to finda feasible solution. Interestingly, similarly faithful em-beddings were obtained in shorter times using as fewas m=10 landmarks, though the input reconstructionsin these cases were of considerably worse quality. Alsoworth mentioning is that adding low variance Gaus-sian noise to the inputs had no significant impact onthe algorithm’s performance.

The second data set was created from the n = 2000most common words in the ARPA North AmericanBusiness News corpus. Each of these words was repre-sented by its discrete probability distribution over theD = 60000 words that could possibly follow it. Thedistributions were estimated from a maximum likeli-hood bigram model. The embedding of these high di-mensional distributions was performed by `SDE (withk = 4, r = 12, and m = 30) in about 35 minutes; thevariance of the embedding, as revealed by the eigen-value spectrum of the landmark kernel matrix, was es-sentially confined to d=5 dimensions. Table 1 showsa selection of words and their four nearest neighborsin the low dimensional embedding. Despite the mas-sive dimensionality reduction from D=60000 to d=5,many semantically meaningful neighborhoods are seento be preserved.

The third experiment was performed on n=400 colorimages of a teapot viewed from different angles in theplane. Each vectorized image had a dimensionality ofD=23028, resulting from 3 bytes of color informationfor each of 76×101 pixels. In previous work [22] itwas shown that SDE represents the angular mode ofvariability in this data set by an almost perfect cir-cle. Fig. 2 compares embeddings from `SDE (k = 4,r = 12, m = 20) with normal SDE (k = 4) and LLE(r = 12). The eigenvalue spectrum of `SDE is very

Figure 3: Top: Error rate of five-nearest-neighborsclassification on the test set of USPS handwritten dig-its. The error rate is plotted against the dimensional-ity of embeddings from PCA and `SDE (with k = 4,r =12, m=10). It can be seen that `SDE preserves theneighborhood structure of the digits fairly well withonly a few dimensions. Bottom: Normalized eigen-value spectra from `SDE and PCA. The latter revealsmany more dimensions with appreciable variance.

similar to that of SDE, revealing that the variance ofthe embedding is concentrated in two dimensions. Theresults from `SDE do not exactly reproduce the re-sults from SDE on this data set, but the differencebecomes smaller with increasing number of landmarks(at the expense of more computation time). Actu-ally, as shown in Fig. 4, `SDE (which took 79 seconds)is slower than SDE on this particular data set. Theincrease in computation time has two simple explana-tions that seem peculiar to this data set. First, thisdata set is rather small, and `SDE incurs some over-head in its setup that is only negligible for large datasets. Second, this data set of images has a particularcyclic structure that is easily “broken” if the moni-tored constraints are not sampled evenly. Thus, thisparticular data set is not well-suited to the incremen-tal scheme for adding unenforced constraints in `SDE;a large number of SDP reruns are required, resultingin a longer overall computation time than SDE. (SeeTable 2.)

The final experiment was performed on the entire dataset of n=9298 USPS handwritten digits [10]. The in-puts were 16×16 pixel grayscale images of the scanneddigits. Table 2 shows that only 690 out of 61735 in-equality constraints needed to be explicitly monitoredby the SDP solver for `SDE to find a feasible solution.This made it possible to obtain an embedding in 40minutes (with k = 4, r = 12, m = 10), whereas earlierimplementations of SDE could not handle problems

Figure 4: Relative speedup of `SDE versus SDE ondata sets with different numbers of examples (n) andlandmarks (m). Speedups of two orders of magnitudeare observed on larger data sets. On small data sets,however, SDE can be faster than `SDE.

of this size. To evaluate the embeddings from `SDE,we compared their nearest neighbor classification errorrates to those of PCA. The top plot in Fig. 3 shows theclassification error rate (using five nearest neighborsin the training images to classify test images) versusthe dimensionality of the embeddings from `SDE andPCA. The error rate from `SDE drops very rapidlywith dimensionality, nearly matching the error rate onthe actual images with only d=3 dimensions. By con-trast, PCA requires d=12 dimensions to overtake theperformace of `SDE. The bar plot at the bottom ofFig. 3 shows the normalized eigenvalue spectra fromboth `SDE and PCA. From this plot, it is clear that`SDE concentrates the variance of its embedding inmany fewer dimensions than PCA.

When does `SDE outperform SDE? Figure 4 shows thespeedup of `SDE versus SDE on several data sets. Notsurprisingly, the relative speedup grows in proportionwith the size of the data set. Small data sets (withn < 500) can generally be unfolded faster by SDE,while larger data sets (with 500 < n < 2000) can beunfolded up to 400 times faster by `SDE. For evenlarger data sets, only `SDE remains a viable option.

5 Conclusion

In this paper, we have developed a much faster algo-rithm for manifold learning by semidefinite program-ming. There are many aspects of the algorithm thatwe are still investigating, including the interplay be-tween the number and placement of landmarks, thedefinition of local neighborhoods, and the quality of

Figure 2: Comparison of embeddings from SDE, LLE and `SDE for n = 400 color images of a rotating teapot.The vectorized images had dimension D=23028. LLE (with r=12) and `SDE (with k=4, r=12, m=20) yieldsimilar but slightly more irregular results than SDE (with k=4). The normalized eigenspectra in SDE and `SDE(i.e., the eigenspectra divided by the trace of their kernel matrices) reveal that the variances of their embeddingsare concentrated in two dimensions; the eigenspectrum from LLE does not reveal this sort of information.

data set n m constraints monitored time (secs)teapots 400 20 1599 565 79bigrams 2000 30 11170 1396 2103

USPS digits 9298 10 61735 690 2420Swiss roll 10000 20 43182 1205 968

Table 2: Total number of constraints versus number of constraints explicitly monitored by the SDP solver for`SDE on several data sets. The numbers of inputs (n) and landmarks (m) are also shown, along with computationtimes. The speedup of `SDE is largely derived from omitting redundant constraints.

the resulting reconstructions and embeddings. Nev-ertheless, our initial results are promising and showthat manifold learning by semidefinite programmingcan scale to much larger data sets than we originallyimagined in earlier work [21, 22].

Beyond the practical applications of `SDE, the frame-work in this paper is interesting in the way it com-bines ideas from several different lines of recent work.`SDE is based on the same appeals to symmetry atthe heart of LLE [13, 14] and SDE [21, 22]. Thelinear reconstructions that yield the factorization ofthe kernel matrix in eq. (1) are also reminiscent ofsemi-supervised algorithms for propagating labeled in-formation through large graphs of unlabeled exam-ples [1, 16, 18, 26, 27]. Finally, though based on asomewhat different intuition, the computational gainsof `SDE are similar to those obtained by landmarkmethods for Isomap [7].

While we have applied `SDE (in minutes) to datasets with as many as n = 10000 examples, there ex-ist many larger data sets for which the algorithm re-mains impractical. Further insights are therefore re-quired. In related work, we have developed a simpleout-of-sample extension for SDE, analogous to similar

extensions for other spectral methods [3]. Algorithmicadvances may also emerge from the dual formulation of“maximum variance unfolding” [17], which is relatedto the problem of computing fastest mixing Markovchains on graphs. We are hopeful that a combinationof complementary approaches will lead to even fasterand more powerful algorithms for manifold learning bysemidefinite programming.

Acknowledgments

We are grateful to Ali Jadbabaie (University of Penn-sylvania) for several discussions about semidefiniteprogramming and to the anonymous reviewers formany useful comments.

References

[1] M. Belkin, I. Matveeva, and P. Niyogi. Regulariza-tion and semi-supervised learning on large graphs. InProceedings of the Seventeenth Annual Conference onComputational Learning Theory (COLT 2004), pages624–638, Banff, Canada, 2004.

[2] M. Belkin and P. Niyogi. Laplacian eigenmaps for di-mensionality reduction and data representation. Neu-ral Computation, 15(6):1373–1396, 2003.

[3] Y. Bengio, J-F. Paiement, and P. Vincent. Out-of-sample extensions for LLE, Isomap, MDS, eigenmaps,and spectral clustering. In S. Thrun, L. K. Saul, andB. Scholkopf, editors, Advances in Neural InformationProcessing Systems 16, Cambridge, MA, 2004. MITPress.

[4] B. Borchers. CSDP, a C library for semidefiniteprogramming. Optimization Methods and Software11(1):613-623, 1999.

[5] M. Brand. Charting a manifold. In S. Becker,S. Thrun, and K. Obermayer, editors, Advances inNeural Information Processing Systems 15, pages 985–992, Cambridge, MA, 2003. MIT Press.

[6] N. Cristianini and J. Shawe-Taylor. An Introductionto Support Vector Machines. Cambridge UniversityPress, Cambridge, UK, 2000.

[7] V. de Silva and J. B. Tenenbaum. Global versus lo-cal methods in nonlinear dimensionality reduction. InS. Becker, S. Thrun, and K. Obermayer, editors, Ad-vances in Neural Information Processing Systems 15,pages 721–728, Cambridge, MA, 2003. MIT Press.

[8] D. L. Donoho and C. E. Grimes. Hessian eigen-maps: locally linear embedding techniques for high-dimensional data. Proceedings of the NationalAcademy of Arts and Sciences, 100:5591–5596, 2003.

[9] J. Ham, D. D. Lee, S. Mika, and B. Scholkopf. A kernelview of the dimensionality reduction of manifolds. InProceedings of the Twenty First International Confer-ence on Machine Learning (ICML-04), pages 369–376,Banff, Canada, 2004.

[10] J. J. Hull. A database for handwritten text recognitionresearch. IEEE Transaction on Pattern Analysis andMachine Intelligence, 16(5):550–554, May 1994.

[11] J. C. Platt. Fast embedding of sparse similaritygraphs. In S. Thrun, L. K. Saul, and B. Scholkopf,editors, Advances in Neural Information ProcessingSystems 16, Cambridge, MA, 2004. MIT Press.

[12] J. C. Platt. FastMap, MetricMap, and landmark MDSare all nystrom algorithms. In Proceedings of the TenthInternational Workshop on Artificial Intelligence andStatistics, Barbados, WI, January 2005.

[13] S. T. Roweis and L. K. Saul. Nonlinear dimension-ality reduction by locally linear embedding. Science,290:2323–2326, 2000.

[14] L. K. Saul and S. T. Roweis. Think globally, fit lo-cally: unsupervised learning of low dimensional man-ifolds. Journal of Machine Learning Research, 4:119–155, 2003.

[15] B. Scholkopf, A. J. Smola, and K.-R. Muller. Nonlin-ear component analysis as a kernel eigenvalue prob-lem. Neural Computation, 10:1299–1319, 1998.

[16] A. J. Smola and R. Kondor. Kernels and regulariza-tion on graphs. In Proceedings of the Sixteenth An-nual Conference on Computational Learning Theoryand Kernel Workshop, Washington D.C., 2003.

[17] J. Sun, S. Boyd, L. Xiao, and P. Diaconis. The fastestmixing Markov process on a graph and a connectionto a maximum variance unfolding problem. SIAM Re-view, submitted.

[18] M. Szummer and T. Jaakkola. Partially labeled clas-sification with Markov random walks. In T. G. Di-etterich, S. Becker, and Z. Ghahramani, editors, Ad-vances in Neural Information Processing Systems 14,Cambridge, MA, 2002. MIT Press.

[19] J. B. Tenenbaum, V. de Silva, and J. C. Langford. Aglobal geometric framework for nonlinear dimension-ality reduction. Science, 290:2319–2323, 2000.

[20] L. Vandenberghe and S. P. Boyd. Semidefinite pro-gramming. SIAM Review, 38(1):49–95, March 1996.

[21] K. Q. Weinberger and L. K. Saul. Unsupervisedlearning of image manifolds by semidefinite program-ming. In Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition (CVPR-04), volume 2, pages 988–995, Washington D.C., 2004.

[22] K. Q. Weinberger, F. Sha, and L. K. Saul. Learninga kernel matrix for nonlinear dimensionality reduc-tion. In Proceedings of the Twenty First InternationalConference on Machine Learning (ICML-04), pages839–846, Banff, Canada, 2004.

[23] C. K. I. Williams. On a connection between kernelPCA and metric multidimensional scaling. In T. K.Leen, T. G. Dietterich, and V. Tresp, editors, Ad-vances in Neural Information Processing Systems 13,pages 675–681, Cambridge, MA, 2001. MIT Press.

[24] Christopher K. I. Williams and Matthias Seeger. Us-ing the Nystrom method to speed up kernel machines.In T. Leen, T. Dietterich, and V. Tresp, editors, Neu-ral Information Processing Systems 13, pages 682–688, Cambridge, MA, 2001. MIT Press.

[25] Z. Zhang and H. Zha. Principal manifolds and non-linear dimensionality reduction by local tangent spacealignment. SIAM Journal of Scientific Computing, inpress.

[26] D. Zhou, O. Bousquet, T. N. Lai, J. Weston, andB. Scholkopf. Learning with local and global consis-tency. In S. Thrun, L. K. Saul, and B. Scholkopf, edi-tors, Advances in Neural Information Processing Sys-tems 16, pages 321–328, Cambridge, MA, 2004. MITPress.

[27] X. Zhu, Z. Ghahramani, and J. Lafferty. Semi-supervised learning using Gaussian fields and har-monic functions. In Proceedings of the Twentieth In-ternational Conference on Machine Learning (ICML2003), pages 912–919, Washington D.C., 2003.