ICML2016: Low-rank tensor completion: a Riemannian manifold preconditioning approach

Introduction Metric proposal Tucker manifold geometry Numerical Comparisons

Low-rank tensor completion:a Riemannian manifold preconditioning approach

Hiroyuki Kasai † and Bamdev Mishra††

†The University of Electro-Communications, Japan

††Amazon Development Centre India, India

ICML2016, New York, USA, June 20, 2016

Low-rank tensor completion: a Riemannian manifold preconditioning approach (ICML2016) (copyrights by Kasai & Mishra) 1


Tensor completion problem

Goal: estimate entries of unknown or missing values.

Matrix completion problem Tensor completion problem

Applications: collaborative filtering, signal recovery, etc.



Our contributions (Summary)

Address a Riemannian optimization framework.

Propose a novel Riemannian metric by exploiting thesymmetry of constraints and the structure of the cost.(Riemannian manifold preconditioning)

Develop concrete matrix expressions and algorithms with thisnovel metric.

Show superior performances of our proposed algorithms forlow-sampling and ill-conditioned large-scale data (1012).



Mathematical problem definition

3-order tensors X ∈ Rn1×n2×n3 is addressed hereafter.

Definition (Tensor completion problem with fixed-rank)

minX∈Rn1×n2×n3

1

|Ω|∥PΩ(X )−PΩ(X ⋆)∥2F

subject to rank(X ) = r ,(1)

PΩ(·) extracts only elements in the observed set Ω.

rank(X ) = r = (r1, r2, r3) is multilinear rank of X .

Higher scalability to huge-data due to NO expensive SVDthan nuclear norm regularization approaches.

Assume a low-rank structure by Tucker decomposition.



Tucker decomposition and Tucker manifold

Tucker decomposition of rank r (=(r1, r2, r3)) is[Kolda and Bader, 2009]

X = G×1U1×2U2×3U3 (∈ Rn1×n2×n3)

Ud ∈ St(rd, nd), i.e., Stiefel manifold of matrices of sizend × rd with orthogonal columns, and G ∈ Rr1×r2×r3 .

Tucker manifold M is defined as

M := St(r1, n1)× St(r2, n2)× St(r3, n3)× Rr1×r2×r3 .Low-rank tensor completion: a Riemannian manifold preconditioning approach (ICML2016) (copyrights by Kasai & Mishra) 5


The quotient structure of Tucker decomposition

Solutions are not isolated, but equivalence classes due toinvariant property for Od ∈ O(rd),

[(U1,U2,U3,G)] :=

(U1O1,U2O2,U3O3,G×1OT1 ×2O

T2 ×3O

T3 ) : Od ∈ O(rd).

Quotient manifold M/∼

M/∼ := M/(O(r1)×O(r2)×O(r3)),

⇓

Solve (1) as an unconstrained optimization problem on aRiemannian quotient manifold by endowing M/∼.



The least-squares structure of the cost function

Problem (1) is convex and quadratic in X and(U1,U2,U3,G) individually.

⇓ Address the block diagonal approximation of ∥X −X ⋆∥2F for

a new metric.

((G1GT1 )⊗ In1 , (G2G

T2 )⊗ In2 , (G3G

T3 )⊗ In3 , Ir1r2r3).

As a result, exploit second-order information in first-orderalgorithms.



Propose a novel Riemannian metric

Propose a metric from symmetry and least-squares structure.

Definition (New Riemannian metric)

gx : TxM× TxM → R is defined as

gx(ξx, ηx) = ⟨ξU1 , ηU1(G1GT1 )⟩+ ⟨ξU2 , ηU2(G2G

T2 )⟩

+⟨ξU3 , ηU3(G3GT3 )⟩+ ⟨ξG , ηG⟩,

ξx = (ξU1, ξU2

, ξU3, ξG) and ηx = (ηU1

, ηU2, ηU3

, ηG) ∈TxM are tangent vectors.

gx(ξx, ηx) is invariant to [x] (= [(U1,U2,U3,G)]).



Geometry of quotient manifold



Geometry of Tucker manifold

Show concrete mathematical expressions. Propose a preconditioned nonlinear conjugate gradient.



Numerical comparisons

We compare with a number of state-of-the-art algorithms undervarious scenarios.

Tucker decomposition based algorithms; TOpt [Filipovic and Jukic, 2013], geomCG

[Kressner et al., 2014]

Nuclear norm minimization algorithms; HaLRTC [Liu et al., 2013], Latent [Tomioka et al., 2011],

Hard [Signoretto et al., 2014]



Results on synthetic and real datasets (1)

10 20 30 10 20 30 10 20 3010

0

101

102

103

104

100 × 100 × 100

150 × 150 × 150

200 × 200 × 200

Tim

e in

sec

on

ds

OS

ProposedgeomCGHaLRTCTOptLatentHard

(a) Small scale

10 20 30 10

−15

10−10

10−5

100

105

OSM

ean

sq

uar

e er

ror

on

Γ

Proposed(100x100x100)geomCG(100x100x100)HalLRTC(100x100x100)TOpt(100x100x100)Latent(100x100x100)Hard(100x100x100)Proposed(150x150x150)geomCG(150x150x150)HalLRTC(150x150x150)TOpt(150x150x150)Latent(150x150x150)Hard(150x150x150)Proposed(200x200x200)geomCG(200x200x200)HalLRTC(200x200x200)TOpt(200x200x200)Latent(200x200x200)Hard(200x200x200)

(b) Small scale

3000 5000 10000 3000 5000 1000010

0

101

102

103

104

Tensor Size (per dimension)

Tim

e in

sec

on

ds

r=(5,5,5)

r=(10,10,10)ProposedgeomCG

(c) Large scale.

0 50 100 150 20010

−4

10−2

100

102

Time in seconds

Mea

n s

qu

are

erro

r o

n Γ

Proposed (CN=5)geomCG (CN=5)Proposed (CN=50)geomCG (CN=50)Proposed (CN=100)geomCG (CN=100)

(f) Ill-cond. & low sample

(5,5,5) (5,5,5) (5,5,5) (7,6,6) (10,5,5)(15,4,4)0

100

200

300

400

500

600

700

rank

Tim

e in

sec

on

ds

(a)20000× 7000× 7000

30000× 6000× 6000

40000× 5000× 5000

(b) 10000 × 10000× 10000

ProposedgeomCG

(h) Rectangular instance

0 100 200 30010

−4

10−2

100

102

104

Time in secondsM

ean

sq

uar

e er

ror

on

Γ

ProposedgeomCGHaLRTCTOptLatentHard

(i) hyperspectral image.



Results on synthetic and real datasets (2)

Hyperspectral imageRibeira OS = 11 OS = 22Algorithm Time MSE on Γ Time MSE on Γ

Proposed 33 ± 13 8.2095 · 10−4 ± 1.7 · 10−5 67 ± 43 6.9516 · 10−4 ± 1.1 · 10−5

geomCG 36 ± 14 3.8342 · 10−1 ± 4.2 · 10−2 150 ± 48 6.2590 · 10−3 ± 4.5 · 10−3

HaLRTC 46 ± 0 2.2671 · 10−3 ± 3.6 · 10−5 48 ± 0 1.3880 · 10−3 ± 2.7 · 10−5

TOpt 80 ± 32 1.7854 · 10−3 ± 3.8 · 10−4 27 ± 21 2.1259 · 10−3 ± 3.8 · 10−4

Latent 553 ± 3 2.9296 · 10−3 ± 6.4 · 10−5 558 ± 3 1.6339 · 10−3 ± 2.3 · 10−5

Hard 400 ± 5 6.5090 · 102 ± 6.1 · 101 402 ± 4 6.5989 · 102 ± 9.8 · 101

MovieLens-10MMovieLens Proposed geomCG

r Time MSE on Γ Time MSE on Γ

(4, 4, 4) 1748 ± 441 0.6762 ± 1.5 · 10−3 2981 ± 40 0.6956±?2.8 · 10−3

(6, 6, 6) 6058 ± 47 0.6913 ± 3.3 · 10−3 6554 ± 655 0.7398±?7.1 · 10−3

(8, 8, 8) 11370 ± 103 0.7589 ± 7.1 · 10−3 13853 ± 118 0.8955±?3.3 · 10−2

(10, 10, 10) 32802 ± 52 1.0107 ± 2.7 · 10−2 38145 ± 36 1.6550±?8.7 · 10−2



Conclusions

Tackled a low-rank tensor completion problem in aRiemannian manifold preconditioning approach.

Proposed a novel Riemannian metric that exploits thefundamental structures of symmetry, due to non-uniqueness ofTucker decomposition, and least-squares of the cost function.

Use the versatile Riemannian optimization framework.

Proposed a preconditioned nonlinear conjugate gradientalgorithm.

Also proposed a stochastic gradient algorithm for online setup.

Concrete matrix expressions are worked out.

Numerical comparisons suggest that our proposed algorithmhas a superior performance on different benchmarks.



Thank you for your attention

Paper ICML paper:

http://jmlr.org/proceedings/papers/v48/kasai16.pdf ICML supplementary paper:

http://jmlr.org/proceedings/papers/v48/kasai16-supp.pdf

Software (Matlab codes) Independent project:

http://bamdevmishra.com/codes/tensorcompletion/ Built-in project in Manopt:

http://www.manopt.org/fixedrankfactory_tucker_

preconditioned


http://bamdevmishra.com/codes/tensorcompletion/

http://www.manopt.org/ fixedrankfactory_tucker_preconditioned

http://www.manopt.org/ fixedrankfactory_tucker_preconditioned


References I

Absil, P.-A., Mahony, R., and Sepulchre, R. (2008).Optimization Algorithms on Matrix Manifolds.Princeton University Press.

Boumal, N., Mishra, B., Absil, P.-A., and Sepulchre, R. (2014).Manopt: a Matlab toolbox for optimization on manifolds.J. Mach. Learn. Res., 15(1):1455–1459.

Filipovic, M. and Jukic, A. (2013).Tucker factorization with missing data with application to low-n-rank tensorcompletion.Multidim. Syst. Sign. P.Doi: 10.1007/s11045-013-0269-9.

Foster, D. H., Nascimento, S. M. C., and Amano, K. (2007).Information limits on neural identification of colored surfaces in natural scenes.Visual Neurosci., 21(3):331–336.

Kolda, T. G. and Bader, B. W. (2009).Tensor decompositions and applications.SIAM Review, 51(3):455–500.



References II

Kressner, D., Steinlechner, M., and Vandereycken, B. (2014).Low-rank tensor completion by Riemannian optimization.BIT Numer. Math., 54(2):447–468.

Liu, J., Musialski, P., Wonka, P., and Ye, J. (2013).Tensor completion for estimating missing values in visual data.IEEE Trans. Pattern Anal. Mach. Intell., 35(1):208–220.

Ring, W. and Wirth, B. (2012).Optimization methods on Riemannian manifolds and their application to shapespace.SIAM J. Optim., 22(2):596–627.

Sato, H. and Iwai, T. (2015).A new, globally convergent Riemannian conjugate gradient method.Optimization, 64(4):1011–1031.

Signoretto, M., Dinh, Q. T., Lathauwer, L. D., and Suykens, J. A. K. (2014).Learning with tensors: a framework based on convex optimization and spectralregularization.Mach. Learn., 94(3):303–351.



References III

Tomioka, R., Hayashi, K., and Kashima, H. (2011).Estimation of low-rank tensors via convex optimization.Technical report, arXiv preprint arXiv:1010.0789.



Supplemental slides



Proposal: Invariant property of new Riemannian metric

Proposition (Invariant property of new Riemannian metric)

Let (ξU1 , ξU2 , ξU3 , ξG) and (ηU1 , ηU2 , ηU3 , ηG) be tangentvectors to the quotient manifold at (U1,U2,U3,G),and (ξU1O1 , ξU2O2 , ξU3O3 , ξG×1O

T1 ×2O

T2 ×3O

T3 )

and (ηU1O1 , ηU2O2 , ηU3O3 , ηG×1OT1 ×2O

T2 ×3O

T3) be

tangent vectors to the quotient manifold at(U1O1,U2O2,U3O3,G×1O

T1 ×2O

T2 ×3O

T3 ). The new

metric (2) are invariant along the equivalence class (6) as

g(U1,U2,U3,G)((ξU1 , ξU2 , ξU3 , ξG), (ηU1 , ηU2 , ηU3 , ηG))

= g(U1O1,U2O2,U3O3,G×1OT1 ×2O

T2 ×3O

T3 )

((ξU1O1 , ξU2O2 , ξU3O3 , ξG×1OT1 ×2O

T2 ×3O

T3),

(ηU1O1 , ηU2O2 , ηU3O3 , ηG×1OT1 ×2O

T2 ×3O

T3)).



Quotient manifold and horizontal lift

x and y belong to the same equivalence class and theyrepresent a single point [x] := y ∈ M : y ∼ x on M/∼.

Tangent space is decomposable into TxM = Vx ⊕Hx, where the vertical space Vx is the tangent space of the equivalence

class [x], and the horizontal space Hx is the orthogonal subspace to Vx.

Vx does NOT induce a displacement along the equivalenceclass [x].

horizontal lift a unique element ξx ∈ Hx that ξ[x] ∈ T[x](M/∼) at [x] has.



Tangent space projection operator (1)Normal space NxM

Obtain from the extracted component of normal space toTxM in the ambient space must locate on the tangent space.

The following lemma can be shown for the normal space Nx;

Lemma (Normal space)

The quotient manifold endowed with the Riemannian metric(2) has the matrix characterization of the normal space NxMdefined as

(U1SU1(G1GT1 )

−1,U2SU2(G2GT2 )

−1,U3SU3(G3GT3 )

−1, 0)

: SUd∈ Rrd×rd ,ST

Ud= SUd

, for d ∈ 1, 2, 3.

where SUdfor all d ∈ 1, 2, 3 are symmetric matrices.



Tangent space projection operator (2)Tangent space projector Ψx

Proposition (Tangent space projection operator)

The quotient manifold endowed with the Riemannian met-ric (2) admits a tangent projector Ψx : Rn1×r1 × Rn2×r2 ×Rn3×r3 × Rr1×r2×r3 → TxM defined as

Ψx(YU1,YU2,YU3,YG) = (YU1−U1SU1(G1GT1 )

−1,

YU2−U2SU2(G2GT2 )

−1,

YU3−U3SU3(G3GT3 )

−1,YG),

where SUdis the solution to the Lyapunov equation below;

SUdGdG

Td + GdG

Td SUd

= GdGTd (Y

TUd

Ud +UTdYUd

)GdGTd

They are solved efficiently with the Matlab’s lyap routine.Low-rank tensor completion: a Riemannian manifold preconditioning approach (ICML2016) (copyrights by Kasai & Mishra) 23


Horizontal space projection operator (1)Vertical space Vx and Horizontal space Hx

This is obtained by Calculate the condition (Lemma below) of the horizontal space

from the orthogonal relationship with the vertical space. Remove the component along the vertical space, and the

remaining components must satisfy the condition above. The vertical space Vx has the matrix characterization

(U1Ω1,U2Ω2,U3Ω3,−(G×1Ω1 + G×2Ω2 + G×3Ω3))

: Ωd ∈ Rrd×rd ,ΩTd = −Ωd for d ∈ 1, 2, 3.

where Ωd for all d ∈ 1, 2, 3 are skew symmetric matrices.

Lemma (Horizontal space)

Horizontal space ξx = (ξU1 , ξU2 , ξU3 , ξG) ∈ Hx must satisfy

(GdGTd )ξ

TUd

Ud + ξGdGT

d is symmetric for d ∈ 1, 2, 3.Low-rank tensor completion: a Riemannian manifold preconditioning approach (ICML2016) (copyrights by Kasai & Mishra) 24


Horizontal space projection operator (2)Vertical space Vx and Horizontal space projector Πx

Proposition (Horizontal space projection operator)

The quotient manifold endowed with the Riemannian metric(2) admits a horizontal projector Πx : TxM :→ Hx : ηx 7→Πx(ηx) defined as

Πx(ηx) = (ηU1−U1Ω1, ηU2−U2Ω2, ηU3−U3Ω3,ηG−(−(G×1Ω1+G×2Ω2+G×3Ω3))),

where ηx = (ηU1 , ηU2 , ηU3 , ηG) ∈ TxM and Ωd is a skew-symmetric matrix of size rd × rd that is the solution to thecoupled Lyapunov equations below;

(continue to the next page.)



Horizontal space projection operator (3)Horizontal space projector Πx

Proposition (Horizontal space projection operator)

(continued from the previous page.)

G1GT1 Ω1 +Ω1G1G

T1 − G1(Ir3 ⊗Ω2)G

T1 − G1(Ω3 ⊗ Ir2)G

T1

= Skew(UT1 ηU1G1G

T1 ) + Skew(G1η

TG1),

G2GT2 Ω2 +Ω2G2G

T2 − G2(Ir3 ⊗Ω1)G

T2 − G2(Ω3 ⊗ Ir1)G

T2

= Skew(UT2 ηU2G2G

T2 ) + Skew(G2η

TG2),

G3GT3 Ω3 +Ω3G3G

T3 − G3(Ir2 ⊗Ω1)G

T3 − G3(Ω2 ⊗ Ir1)G

T3

= Skew(UT3 ηU3G3G

T3 ) + Skew(G3η

TG3).

Skew(·) extracts the skew-symmetric part of a square matrix, i.e.,Skew(D) = (D−DT )/2.

The coupled Lyapunov equations are solved efficiently with theMatlab’s pcg routine that is combined with a specific preconditionerresulting from the Gauss-Seidel approximation.



Retraction Rx(ξx)

Map vectors in the horizontal space to points on the searchspace M and satisfies the local rigidity condition[Absil et al., 2008, Definition 4.1].

Provides a natural way to move on the manifold along asearch direction.

Due to the product nature of M, we can choose a retractionby combining retractions on the individual manifolds, i.e.,

Rx(ξx) = (uf(U1 + ξU1), uf(U2 + ξU2), uf(U3 + ξU3),G + ξG),

where ξx ∈ Hx and uf(·) extracts the orthogonal factor of afull column rank matrix, i.e., uf(A) = A(ATA)−1/2.



Vector transport Tηxξx

Smooth mapping that transports a tangent vector ξx ∈ TxMat x ∈ M to a vector in the tangent space at Rx(ηx)[Absil et al., 2008, Section 8.1.4].

Generalize the classical concept of translation of vectors in theEuclidean space to manifolds.

The horizontal lift of the abstract vector transport Tη[x]ξ[x] onM/∼ has the matrix characterization

ΠRx(ηx)(Tηxξx) = ΠRx(ηx)(ΨRx(ηx)(ξx)),

where ξx and ηx are the horizontal lifts in Hx of ξ[x] and η[x]that belong to T[x](M/∼).



Riemannian gradient computations (1)Horizontal lift (1)

Finally, from the Riemannian submersion theory[Absil et al., 2008, Section 3.6.2], the horizontal lift ofgrad[x]f is equal to gradxf = Ψ(egradxf).

Subsequently, we obtain below;

the horizontal lift of grad[x]f =

(S1(U3 ⊗U2)GT1 (G1G

T1 )

−1 −U1BU1(G1GT1 )

−1,

S2(U3 ⊗U1)GT2 (G2G

T2 )

−1 −U2BU2(G2GT2 )

−1,

S3(U2 ⊗U1)GT3 (G3G

T3 )

−1 −U3BU3(G3GT3 )

−1,

S ×1 UT1 ×2 U

T2 ×3 U

T3 ),



Riemannian gradient computations (2)Horizontal lift (2)

where BUdfor d ∈ 1, 2, 3 are the solutions to the Lyapunov

equationsBU1G1G

T1 + G1G

T1 BU1 = 2Sym(G1G

T1 U

T1 (S1(U3 ⊗U2)G

T1 ),

BU2G2GT2 + G2G

T2 BU2 = 2Sym(G2G

T2 U

T2 (S2(U3 ⊗U1)G

T2 ),

BU3G3GT3 + G3G

T3 BU3 = 2Sym(G3G

T3 U

T3 (S3(U2 ⊗U1)G

T3 ),

which are solved efficiently with the Matlab’s lyap routine.Sym(·) extracts the symmetric part of a square matrix, i.e.,Sym(D) = (D+DT )/2.



Preconditioned Conjugate Gradient (CG) Algorithm

Off-the-shelf conjugate gradient implementation of Manopt 1

[Boumal et al., 2014] for any smooth cost function.

The convergence analysis of the Riemannian conjugategradient method follows from[Sato and Iwai, 2015, Ring and Wirth, 2012].

Cost function specific ingredients are needed; Compute the Riemannian gradient. Compute an initial guess for the step-size for CG.

The total computational cost per iteration of our algorithm isO(|Ω|r1r2r3), where |Ω| is the number of known entries.

1http://www.manopt.org/Low-rank tensor completion: a Riemannian manifold preconditioning approach (ICML2016) (copyrights by Kasai & Mishra) 31


Experiment conditions

Comparisons with state-of-the-art algorithms that include

Tucker decomposition based algorithms; TOpt [Filipovic and Jukic, 2013], geomCG

[Kressner et al., 2014] Nuclear norm minimization algorithms;

HaLRTC [Liu et al., 2013], Latent [Tomioka et al., 2011],Hard [Signoretto et al., 2014]

Problem instances

Case S2: small-scale instances. Case S3: large-scale instances. Case S5: influence of ill-conditioning and low sampling. Case S7: asymmetric instances. Case R1: hyperspectral image “Ribeira” [Foster et al., 2007]. Case R2: MovieLens-10M.

Note that, compared with ONLY geomCG for large-scaleinstances, i.e., except Case S2.



Synthetic dataset cases

Case S2: small-scale instances. Size 100× 100× 100, 150× 150× 150, and 200× 200× 200

and rank r = (10, 10, 10) are considered. OS is 10, 20, 30. Case S3: large-scale instances.

Tensors of size 3000× 3000× 3000, 5000× 5000× 5000, and10000× 10000× 10000 and ranks r = (5, 5, 5) and(10, 10, 10). OS is 10.

Case S5: influence of ill-conditioning and low sampling. Case S4 with OS = 5. We impose a diagonal core G with

exponentially decaying positive values of condition numbers(CN) 5, 50, and 100.

Case S7: asymmetric instances. (a): tensors size 20000× 7000× 7000, 30000× 6000× 6000,

and 40000× 5000× 5000 with rank r = (5, 5, 5). (b): tensor size 10000× 10000× 10000 with ranks (7, 6, 6),

(10, 5, 5), and (15, 4, 4).Low-rank tensor completion: a Riemannian manifold preconditioning approach (ICML2016) (copyrights by Kasai & Mishra) 33


Real-world dataset cases

Case R1: hyperspectral image “Ribeira”.[Foster et al., 2007]

The tensor size is 203× 268× 33. Compare all the algorithms, and perform five random samplings

of the pixels based on the OS values 11 and 22, correspondingto the rank r=(15, 15, 6) adopted in [Kressner et al., 2014].

While OS = 22 corresponds to the observation ratio of 10%studied in [Kressner et al., 2014], OS = 11 considers achallenging scenario with the observation ratio of 5%.

Case R2: MovieLens-10M2. This dataset contains 10000054 ratings corresponding to

71567 users and 10681 movies. Split the time into 7-days wide bins results, and finally, get a

tensor of size 71567× 10681× 731. The fraction of known entries is less than 0.002%.

2http://grouplens.org/datasets/movielens/.Low-rank tensor completion: a Riemannian manifold preconditioning approach (ICML2016) (copyrights by Kasai & Mishra) 34

http://grouplens.org/datasets/movielens/

Date post:	11-Apr-2017
Category:	Data & Analytics
Upload:	hiroyuki-kasai
View:	155 times
Download:	0 times

ICML2016: Low-rank tensor completion: a Riemannian manifold preconditioning approach

Data & Analytics