A User's Guide to Riemannian Newton-Type Methods on...

Motivation Outline

A User’s Guide to Riemannian Newton-TypeMethods on Manifolds

Felipe Álvarez

Departamento de Ingeniería MatemáticaCentro de Modelamiento Matemático (CNRS UMI 2807)

Universidad de Chile

In collaboration with: J. Bolte, J. Munier, J. López

Sixièmes Journées Franco-Chiliennes d’OptimisationUniversité du Sud Toulon-Var

Mai 19-21, 2008

http://www.dim.uchile.cl/~falvarez

Felipe Álvarez (DIM) A User’s Guide to Riemannian Newton-Type Methods 1/23

Motivation Outline

Motivation: Nonlinear equations in a manifold

Goal: find p∗ ∈ M satisfying F (p∗) = 0 ∈ Tp∗M

M is a connected and n-dimensional differentiable manifold.

TpM ' Rn is the tangent space of M at p:

If c(t) is a curve passing through p at t = 0 then c(0) ∈ TpM.

F : M → TM is a continuously differentiable vector field:

M 3 p 7→ F (p) ∈ TpM


Motivation Outline







M 3 p 7→ F (p) ∈ TpM


Motivation Outline







M 3 p 7→ F (p) ∈ TpM


Motivation Outline







M 3 p 7→ F (p) ∈ TpM


Motivation Outline

Example 1: Rayleigh’s quotient on the sphere

M = Sn = x ∈ Rn+1 | xT x = 1 (unit sphere in Rn+1).

TxM = v ∈ Rn+1 | xT v = 0.

F (x) = Ax − q(x)x with

A ∈ Rn×n being symmetric and positive definite. q(x) = xT Ax .

xT F (x) = 0⇒ F (x) ∈ TxM.

F (x∗) = 0 iff x∗ is an eigenvector of A with q(x∗) thecorresponding eigenvalue.


Motivation Outline

Example 2: Stiefel manifold

M = Sn,k = Y ∈ Rn×k | Y T Y = Ik.

TY Sn,k = ∆ ∈ Rn×k | ∆T Y + Y T ∆ = 0.

If k = 1 then Sn,1 = Sn−1.

If k = n then Sn,n = On the orthogonal group.

TIn On = ∆ ∈ Rn×n | ∆T = −∆.

dim Sn,k = nk − 12 k(k + 1).

F (Y ) = AY − YY T AY

F (Y ∗) = 0 iff the columns of Y ∗ are eigenvectors of A.


Motivation Outline


M = Sn,k = Y ∈ Rn×k | Y T Y = Ik.

TY Sn,k = ∆ ∈ Rn×k | ∆T Y + Y T ∆ = 0.



TIn On = ∆ ∈ Rn×n | ∆T = −∆.

dim Sn,k = nk − 12 k(k + 1).




Motivation Outline


M = Sn,k = Y ∈ Rn×k | Y T Y = Ik.

TY Sn,k = ∆ ∈ Rn×k | ∆T Y + Y T ∆ = 0.



TIn On = ∆ ∈ Rn×n | ∆T = −∆.

dim Sn,k = nk − 12 k(k + 1).




Motivation Outline


M = Sn,k = Y ∈ Rn×k | Y T Y = Ik.

TY Sn,k = ∆ ∈ Rn×k | ∆T Y + Y T ∆ = 0.



TIn On = ∆ ∈ Rn×n | ∆T = −∆.

dim Sn,k = nk − 12 k(k + 1).




Motivation Outline

Solving nonlinear equations: Euclidean case

Goal: find p∗ ∈ Ω such that F (p∗) = 0 ∈ Rn , where Ω is open

and F : Ω ⊂ Rn → Rn is a C1 vector field.

Newton’s method: F (pk ) + F ′(pk )(pk+1 − pk ) = 0.

−1 −0.5 0 0.5 1 1.5 2 2.5−4

−2

0

2

4

6

8

10

pk+1

pk p*


Motivation Outline

Outline

1 Abstract differential geometry setting for R-Newton

2 Other explicit examples


Abstract differential geometry setting for R-Newton Other explicit examples

Outline





Metric framework

M is endowed with a Riemannian metric g:

‖v‖2p = g(p)(v , v) for v ∈ TpM.

Riemannian distance d : M ×M → [0,+∞):

d(p,q) = inf∫ b

a‖c(t)‖c(t)dt | c : [a,b]→ M, c(a) = p, c(b) = q

Assumption:

(M,d) is a complete metric space.

Covariant derivative:

F ′(p)v := ∇v F (p) = (∇Y F )(p), v ∈ TpM ,

where Y is any vector field on M satisfying v = Y (p). ∇ is the Riemannian (or Levi-Civita) connection on (M,g).



Metric framework




d(p,q) = inf∫ b


Assumption:



F ′(p)v := ∇v F (p) = (∇Y F )(p), v ∈ TpM ,




Metric framework




d(p,q) = inf∫ b


Assumption:



F ′(p)v := ∇v F (p) = (∇Y F )(p), v ∈ TpM ,




Metric framework




d(p,q) = inf∫ b


Assumption:



F ′(p)v := ∇v F (p) = (∇Y F )(p), v ∈ TpM ,




Metric framework




d(p,q) = inf∫ b


Assumption:



F ′(p)v := ∇v F (p) = (∇Y F )(p), v ∈ TpM ,




Exponential MapGeodesic: a curve γ : (a,b)→ M with ∇γ γ = 0.If γi are the coordinates of γ,

d2γk

dt2 +n∑

i,j=1

Γkij

dγi

dtdγj

dt= 0; k = 1, . . . ,n,

where Γki,j are the Christoffel symbols.

Exponential map: expp : TpM → M is defined by setting

expp[v ] = γ(1),

where γ : R→ M is the geodesic with γ(0) = p and γ(0) = v .




d2γk

dt2 +n∑

i,j=1

Γkij

dγi

dtdγj

dt= 0; k = 1, . . . ,n,



expp[v ] = γ(1),





d2γk

dt2 +n∑

i,j=1

Γkij

dγi

dtdγj

dt= 0; k = 1, . . . ,n,



expp[v ] = γ(1),




Riemannian Newton’s method (Shub ’86)

1 Data: Given pk with F (pk ) 6= 0 and F ′(pk ) nondegenerate.2 Newton’s correction: Find vk ∈ Tpk M s.t. F ′(pk )vk = −F (pk ).

3 Update: Set pk+1 = exppk[vk ] .

pk

pk+1

Tp

k

Sn

−X’(pk)−1X(p

k)



Rayleigh quotient (continuation)

Manifold: M = Sn = x ∈ Rn+1 | xT x = 1 (unit sphere in Rn+1).

Tangent space: TxSn = v ∈ Rn+1 | xT v = 0.Metric: g(v ,w) = vT w .

Exponential map:

expp[v ] = p cos(‖v‖) +v‖v‖

sin(‖v‖)

Vector field: F (x) = Ax − q(x)x with q(x) = xT Ax .

Newton direction at p:

v = −p +1

pT ww

where

(A− q(p)I)w = p






Exponential map:


sin(‖v‖)



v = −p +1

pT ww

where

(A− q(p)I)w = p






Exponential map:


sin(‖v‖)



v = −p +1

pT ww

where

(A− q(p)I)w = p






Exponential map:


sin(‖v‖)



v = −p +1

pT ww

where

(A− q(p)I)w = p




A =

0BBBBBB@1 1 1 1 1 11 2 3 4 5 61 3 6 10 15 211 4 10 20 35 561 5 15 35 70 1261 6 21 56 126 252

1CCCCCCA

1 2 3 4 5 6 70

20

40

60

80

100

120

140

160

Iteration

||Ax−

(xT A

x)x|

|



Some references

UDRISTE ’94 (VOL. 297, KLUWER): Convexity and optimization onmanifolds, including R-Newton.

SMITH ’94 (FIELDS INSTITUTE COMM.): Existence of a basin ofattraction for quadratic convergence.

EDELMAN, ARIAS & SMITH ’98 (SIAM J. ON MATRIX ANAL. ANDAPPL): Matrix orthogonality constrains.

FERREIRA & SVAITER ’02 (J. OF COMPLEXITY): Kantorovich-typeproximity test for quadratic convergence under a Lipschitz condition.

DEDIEU, PRIOURET & MALAJOVICH ’03 (IMA J.NUMER. ANAL.):Smale-type proximity test for analytic vector fields.

A., BOLTE & MUNIER (FOUND. COMP. MATHEMATICS ’08): Generaland unifying proximity test.



Outline





Positivity constraints

M = Rn++ and g(p)(u, v) =

∑nk=1 uk vk/h2

k (pk ) wherehi : R++ → R++ differentiable function.

Isometry: γ = γ(t) geodesic ⇐⇒

solution∫ 1

hi (γk ) dγk = ak t + bk

For the barrier φ(p) = −∑n

k=1 log(pk ),∇2φ(p) =diag(h2

1(p1), . . . ,h2n(pn)) where hk (pk ) = pk .

Thus,

γ geodesicsγ(0) = p, γ(0) = v

⇐⇒ γk (t) = pk exp(t vk

pk)





∑nk=1 uk vk/h2



solution∫ 1





Thus,



pk)





∑nk=1 uk vk/h2



solution∫ 1





Thus,



pk)





∑nk=1 uk vk/h2



solution∫ 1





Thus,



pk)



Relative interior of the unitary simplex

M = ∆n−1++ = p ∈ Rn |

n∑i=1

pi = 1, pi > 0, i = 1, . . . ,n,

Tangent space: Tp∆n−1 = v ∈ Rn |n∑

i=1vi = 0

Metric: g(p)(u, v) = (1− 1n )

n∑k=1

uk vkh2

k (pk ).

γ = γ(t) geodesics ⇐⇒

solutionddt

(1

hk (γk )dγkdt −

1n

∑ni=1

1hi (γi )

dγidt

)= 0

For hk (pk ) = pk


⇐⇒ γk (t) =

pk exp(t vkpk

)∑ni=1 pi exp(t vi

pi)




M = ∆n−1++ = p ∈ Rn |

n∑i=1

pi = 1, pi > 0, i = 1, . . . ,n,


i=1vi = 0

Metric: g(p)(u, v) = (1− 1n )

n∑k=1

uk vkh2

k (pk ).


solutionddt

(1

hk (γk )dγkdt −

1n

∑ni=1

1hi (γi )

dγidt

)= 0

For hk (pk ) = pk


⇐⇒ γk (t) =

pk exp(t vkpk


pi)




M = ∆n−1++ = p ∈ Rn |

n∑i=1

pi = 1, pi > 0, i = 1, . . . ,n,


i=1vi = 0

Metric: g(p)(u, v) = (1− 1n )

n∑k=1

uk vkh2

k (pk ).


solutionddt

(1

hk (γk )dγkdt −

1n

∑ni=1

1hi (γi )

dγidt

)= 0

For hk (pk ) = pk


⇐⇒ γk (t) =

pk exp(t vkpk


pi)



The Stiefel manifold (continuation)

Sn,k = A ∈ Rn×k : AT A = Ik

Tangent space: TASn,k = ∆ ∈ Rn×k : ∆T A + AT ∆ = 0

Euclidian Metric: g(∆1,∆2) = trace (∆T1 ∆2)

The equation that described the geodesics is given by

Y (t) + Y (t)(Y (t)T Y (t)) = 0,

and the corresponding exponential map is:

expA[∆] = (A ∆) exp[(

AT ∆ −∆T ∆Ip AT ∆

)]I2p,p exp(−AT ∆),

where I2p,p =

(I2p0p

)and exp(B) = I + B + 1

2 B2 + . . ..



The Stiefel manifold (continuation)

Sn,k = A ∈ Rn×k : AT A = Ik

Tangent space: TASn,k = ∆ ∈ Rn×k : ∆T A + AT ∆ = 0

Euclidian Metric: g(∆1,∆2) = trace (∆T1 ∆2)

The equation that described the geodesics is given by

Y (t) + Y (t)(Y (t)T Y (t)) = 0,

and the corresponding exponential map is:

expA[∆] = (A ∆) exp[(

AT ∆ −∆T ∆Ip AT ∆

)]I2p,p exp(−AT ∆),

where I2p,p =

(I2p0p

)and exp(B) = I + B + 1

2 B2 + . . ..



Stiefel manifold (continuation)Manifold: Sn,k = Y ∈ Rn×k : Y T Y = Ik.

Tangent space: TY Sn,k = ∆ ∈ Rn×k : ∆T Y + Y T ∆ = 0.

Euclidian Metric: g(∆1,∆2) = trace (∆T1 ∆2).

Exponential map:

expY [∆] = (Y ∆) exp[(

Y T ∆ −∆T ∆Ip Y T ∆

)]I2p,p exp(−Y T ∆)

Vector field: V (Y ) = BY − Y (Y T BY )

Newton direction ∆ at Y must satisfy:B∆−∆Y T BY − 1

2 Y (Y T B∆ + ∆T BY + ∆T YY T BY + Y T BYY T ∆) =

−V (Y )






Exponential map:

expY [∆] = (Y ∆) exp[(

Y T ∆ −∆T ∆Ip Y T ∆





−V (Y )






Exponential map:

expY [∆] = (Y ∆) exp[(

Y T ∆ −∆T ∆Ip Y T ∆





−V (Y )






Exponential map:

expY [∆] = (Y ∆) exp[(

Y T ∆ −∆T ∆Ip Y T ∆





−V (Y )



Stiefel Manifold (continuation)

B =

1 1 1 1 11 2 3 4 51 3 6 10 151 4 10 20 351 5 15 35 70

, Y0 =

0.1947 −0.6155 −0.7448−0.4268 0.5660 −0.4579

0.7236 0.1412 0.2122−0.5050 −0.5012 0.3901

0.0355 0.1723 −0.1958

and S5,3.




B =

1 1 1 1 11 2 3 4 51 3 6 10 151 4 10 20 351 5 15 35 70

, Y0 =

0.1947 −0.6155 −0.7448−0.4268 0.5660 −0.4579

0.7236 0.1412 0.2122−0.5050 −0.5012 0.3901

0.0355 0.1723 −0.1958

and S5,3.

1 1.5 2 2.5 3 3.5 4 4.5 50

1

2

3

4

5

6

7

8

Iteration

||BY

−Y

YT B

Y||




B =

1 1 1 1 11 2 3 4 51 3 6 10 151 4 10 20 351 5 15 35 70

, Y ∗ =

0.1795 −0.5882 −0.7500−0.4682 0.6017 −0.4265

0.7062 0.1923 0.2209−0.4847 −0.4777 0.4030

0.1222 0.1635 −0.2108

1 1.5 2 2.5 3 3.5 4 4.5 50

1

2

3

4

5

6

7

8

Iteration

||BY

−Y

YT B

Y||



Cone of Positive Semidefinite MatricesManifold: M = Sn

++ cone of symmetric positive definite matrices.

Tangent space: TPSn++ ' Sn (Sn space symmetric matrices).

Hessian Metric: g(∆1,∆2) = trace(∇2ϕ(P)∆1∆2) with ϕ barrieron Sn

++.Taking ϕ(P) = − log(det(P)): g(∆1,∆2) = trace (P−1∆1P−1∆2)

Goal: minimize f : Sn → R over M

Exponential map:

expP [∆] = P1/2 exp(P−1/2∆P−1/2)P1/2

Vector field: V (P) = P∇f (P)P

Newton direction ∆ at P must satisfy:

V ′(P)∆ = ∇2f (P)∆ + 12 (P−1∆∇f (P) +∇f (P)∆P−1) = −P∇f (P)P









Exponential map:

expP [∆] = P1/2 exp(P−1/2∆P−1/2)P1/2



V ′(P)∆ = ∇2f (P)∆ + 12 (P−1∆∇f (P) +∇f (P)∆P−1) = −P∇f (P)P









Exponential map:

expP [∆] = P1/2 exp(P−1/2∆P−1/2)P1/2



V ′(P)∆ = ∇2f (P)∆ + 12 (P−1∆∇f (P) +∇f (P)∆P−1) = −P∇f (P)P









Exponential map:

expP [∆] = P1/2 exp(P−1/2∆P−1/2)P1/2



V ′(P)∆ = ∇2f (P)∆ + 12 (P−1∆∇f (P) +∇f (P)∆P−1) = −P∇f (P)P



Unitary Generalized Simplex on Sn++

Manifold: M = P ∈ Sn : Trace(P) = 1, P ∈ Sn++ .

Tangent space: TPM = ∆ ∈ Sn : Trace(∆) =∑n

i=1 λi (∆) = 0.

ϕ(P) = − log(det(P)).

Exponential map:

expP [∆] = 1Trace(P exp(P−1/2∆P−1/2))

P1/2 exp(P−1/2∆P−1/2)P1/2



Unitary Generalized Simplex on Sn++

Manifold: M = P ∈ Sn : Trace(P) = 1, P ∈ Sn++ .

Tangent space: TPM = ∆ ∈ Sn : Trace(∆) =∑n

i=1 λi (∆) = 0.

ϕ(P) = − log(det(P)).

Exponential map:

expP [∆] = 1Trace(P exp(P−1/2∆P−1/2))

P1/2 exp(P−1/2∆P−1/2)P1/2



Open problems

Drop out completeness ?

Manifolds with boundary.

Globalization:pk+1 = exppk

[tkvk ]

for some scalar parameter tk > 0.


Date post:	28-Jul-2018
Category:	Documents
Upload:	doankhanh
View:	220 times
Download:	0 times

A User's Guide to Riemannian Newton-Type Methods on...

Documents