Some notes on standard

LM I solvers

Denis Arzelier - Dimitri Peaucelle - Didier Henrion

Laboratoire d’Analyse et d’Architecture des Systemes

LAAS-CNRS

Motivation 2

State-feedback stabilization: x(t) = Ax(t)+Bu(t) and u(t) = Kx(t)

- Quadratic Lyapunov function: v(x(t) = x0P�1x(t)

- Closed-loop Lyapunov equation: (A+BK)P+P(A+BK)0 < 0

- Linearizing change of variable: L = KP

- LM I ’s : AP+PA0+BL+L0B0 < 0 P > 0

A =2

4 �1 0

0 1

35 B =

24 0

1

35 P =

24 p1 p2

p2 p3

35 L =

h

l1 l2

i

Example : 24 �2p1 l1

l1 2p3+2l2

35< 0

24 p1 p2

p2 p3

35> 0

CVUT Praha

04/04/02

Standard LM I problem 3

- Feasibility problem:

9 x 2 Rm j F0+

m

∑i=1

xiFi < 0

| {z }

F(x)

, minx2Rm

λmax(F(x))

- Optimization problem:

min c0x

under

F(x)< 0,

min c0x

under

λmax(F(x))< 0

8 i ; Fi 2 Sn, linearly independant.

CVUT Praha

04/04/02

Standard LM I problem 4

Result 1 :

The LM I problem is a convex and non differentiable optimization problem.

Example : [G. Pataki 00]

F(x) =2

4 �x1�1 �x2

�x2 �1+ x1

35

λmax(F(x)) = 1+q

(x21 + x2

2)

−5

0

5

−5

0

51

2

3

4

5

6

7

8

9

x1

x2

λ max

Nota : [J. Cullum 75], [F. Alizadeh 95] and [Y. Nesterov 94]

CVUT Praha

04/04/02

Semidefinite programming (SDP) 5

Primal problem: Dual problem:

p� = min c0x

under F0 +

m

∑i=1

xiFi < 0

d� = min �Trace(F0Z)

under �Trace(FiZ) = ci i = 1; � � � ;m

Z � 0where x 2 Rm , Z 2 S n, 8 i ; Fi 2 Sm, c 2 Rm .

Applications: automatic control, (robust control), geometric problems with quadratic

forms, statistics, structural design, combinatorial problems, [Nesterov 94],

[Vandenberghe 94], [El Ghaoui 00], [Helmberg 00], [Wolcowicz 00]

CVUT Praha

04/04/02

Some associated problems 6

Second Order Cone Programming : SOCP

min c0x

under jjCix+dijj2 � e0ix+ fi i = 1; � � �;L

where x 2 Rm , Ci 2 R

ni�m, di 2 R

ni , ei 2 R

m , fi 2 R.

Lorentz cone - ice cream cone - quadratic cone: Qni"

Ci

e0i

#

x+"

di

f 0i

#2 Qni =

("

u

t

#j u 2Rni�1; t 2R; jjujj2 � t

)

Embedding of Qni in Sni :

jjujj2 � t ,2

4 t1 u

u0 t

35� 0 and

24 (e0ix+ fi)1 Cix+di

(Cix+di)0 (e0ix+ fi)

35� 0

Applications: [M.S. Lobo 98] grasping force optimization, FIR filter design, truss design.

CVUT Praha

04/04/02

Some associated problems 7

Determinant maximization : MAXDET

min c0x+ log det G(x)�1

under G(x)> 0

F(x)� 0

where x 2 Rm and G : Rm ! Sl , F : Rm ! Sn are affine mappings.

- log det G(x)�1 is related to volume in geometric problems.

- c0x+ log det G(x)�1 is a convex function on fx j G(x)> 0g.

- G(x) = 1, SDP problem.

- c = 0 et F(x) = 1, find the analytical center x� of G(x)> 0.

Applications: minimum volume ellipsoids, matrix completion problems, linear estimation,

Gaussian Channel capacity, [Vandenberghe 96].

CVUT Praha

04/04/02

Some associated problems 8

Generalized eigenvalue problems : GEVP

min λmax(G(x);F(x))

under G(x)> 0

H(x)> 0

where x 2 Rm and G : Rm ! Sl , F : Rm ! Sl , H : Rm ! Sn are affine mappings.

- λmax(G(x);F(x)) is quasi-convex.

- Generalized linear fractional problem:

minc0x+de0x+ f

under Ax+b� 0 ; e0x+ f > 0

Applications: automatic control, linear algebra, [Nesterov 91], [Boyd 94].

CVUT Praha

04/04/02

Summary of key points 9

- Large number of applications:

Automatic control, (robust control).

Truss design.

Mathematic, (geometry, graphs, statistics)

Signal processing, (FIR, antennas).

- Duality theory:

Theoritical analysis.

Algorithms design.

- Algorithmic:

Non differentiable optimization methods.

Interior-point methods.

CVUT Praha

04/04/02

A slice of duality 10

Primal problem: Dual problem:

p� = min c0x

under F0 +

m

∑i=1

xiFi � 0

d� = min �Trace(F0Z)

under �Trace(FiZ) = ci i = 1; � � � ;m

Z � 0

Weak duality: p� � d� and Strong duality: p� = d�

Theorem 1 :

- If primal is strict. feasible with finite p� then p� = d�, reached by dual.

- If dual is strict. feasible with finite d� then p� = d�, reached by primal.

- If primal and dual are strict. feasible then p� = d�.

CVUT Praha

04/04/02

A slice of duality (continued) 11

Nota: Duality gap, conic duality, 6= duality in LP, [Vandenberghe 96], [Ben-Tal 00].

Example : p� = d� = 0 and no solution for the primal

min x1

under

2664

x1� x2

1

x1 + x2

3775 2 Q3

max �λ2

underλ1+λ3 = 1

�λ1+λ3 = 0

λ 2 Q3

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−5

−4

−3

−2

−1

0

1

2

3

4

5

x1

x 2

région primal réalisable

CVUT Praha

04/04/02

A slice of duality (continued) 12

If primal and dual are strictly feasible:

KKT optimality conditions:

Y � 0 Y +F0+

m

∑i=1

xiFi = 0

Z � 0 Trace(FiZ)+ ci = 0 i = 1; � � � ;m

F(x)Z = 0 Complementarity

CVUT Praha

04/04/02

Algorithms, solvers and codes 13

Non differentiable optimization methods: minx2Rm

λmax(F(x))+ c0x

- Convex analysis tools, (subdifferential, cutting planes...).

- Subgradient methods ! bundle methods.

- [Polyak 77], [Shor 85], [Hiriart-Urruty 93], [El Ghaoui 00].

Interior-point methods:

minx2Rm

c0x

F(x)< 0

�! minx2Rm

c0x�µlog det F(x)

- Reduction of the initial problem into a sequence of unconstraint differentiable problems:

self-concordant barrier, Newton methods.

- Centers methods, primal-dual methods and projective method.

- [Fiacco 68], [Karmakar 84], [Nesterov 88], [Nesterov 94], [Alizadeh 95].

CVUT Praha

04/04/02

Primal-dual interior-point methods 14

- Primal-dual central path: (x(µ);Z(µ);µ> 0)

min �trace(F0Z)�µlog det Z

Z > 0

�trace(FiZ) = ci i = 1; � � �;m

Y > 0 ; Y +F0+

m

∑i=1

xiFi = 0

Z > 0 ; Trace(FiZ)+ ci = 0

F(x)Z = µ1

- Newton method

- Simultaneous primal-dual solution

(x(µ);Z(µ)) �! (x�;Z�)

µk ! 0

- 3 directions: XZ (HRVW,KSH,M), NT, XZ+ZX (AHO).

CVUT Praha

04/04/02

A first table 15

ALGO VERSION METHOD MATLAB INPUTS LM I LANG DIR Pb.

SDPA5.0

08.99PIPD YES SDPA YALMIP C++ 3 SDP

SeDuMi 1.05β PIPD YES

DUAL

SDPA

SDPpack

YALMIP

SeDuMiInt

LMILab

MATLAB

C-

SDP

SOCP

CSDP3.0

05.01PIPD NO SDPA YALMIP C HRVW SDP

DSDP33.2

11.00PIDS YES SDPA NO C - SDP

SDPT33.0

05.01PIPD YES

DUAL

SDPA

YALMIP

LMILab

MATLAB

C

HKM

NT

SDP

SOCP

MOSEK2.0

09.01PI YES QPS NO C NT SOCP

LOQO5.04

08.00NLP YES

DUAL

SDPA

AMPL

NO C - SOCP

CVUT Praha

04/04/02

A second table 16

ALGO VERSION METHOD MATLAB INPUTS LM I LANG DIR Pb.

SBMethod1.1.1

11.00BUNDLE NO SBM NO C++ - SDP

SDPHA

(SDPT3)3.0 PIPD YES DUAL

LMITOOL

LMILab

MATLAB

C3 (+4) SDP

SDPpack0.9β

06.97PIPD YES DUAL

LMITOOL

LMILabC AHO

SDP

SQLP

MAXDETα

06.96PIPD YES MATLAB SDPSOL

MATLAB

CNN MAXDET

SP1.1

07.01PIPD YES MATLAB

SDPSOL

YALMIP

LMITOOL

LMILab

MATLAB

CNT SDP

SOCPβ

11.97PIPD YES MATLAB YALMIP

MATLAB

CNN SOCP

LMISOL1.0α

04.97PROJECTIVE NO LMISOL LMISOL LMISOL C SDP

LMILab06.95

PROJECTIVE YES MATLAB LMILabMATLAB

C-

SDP

GEVP

CVUT Praha

04/04/02

Some comparisons 17

Large scale, (nber of variables m and nber of rows n), reliability, (feasibility and accuracy, 6=

measure of errors)), speed, (cpu execution time and memory size), sparsity.

- [H.D. Mittelmann 01] : An independent benchmarking of SDP and SOCP solvers

Comparison of 10 solvers, 8 are available, on 50 problems SDP and SOCP :

Every code is interesting for a sub-class of problems.

Good general codes: SDPA et SeDuMi.

- SDPA, SeDuMi, CSDP, DSDP3, SDPT3, MOSEK, LOQO,

- BMPR, BMZ, BUNDLE

- [Fujisawa 98] : Numerical evaluation of SDPA

Evaluation of SDPA and some comparison between SDPT3, CSDP and SDPSOL

CVUT Praha

04/04/02

References[1] CSDP : http://www.nmt.edu/˜borchers/csdp.html

[2] MAXDET : http://www.stanford.edu/˜boyd/MAXDET/

[3] SDPA : ftp://ftp.is.titech.ac.jp/pup/OpRes/software/SDPA/

[4] SDPT3 : http://www.math.nus.edu.sg/˜mattohkc/

[5] SeDuMi : http://fewcal.kub.nl/sturm/software/sedumi.html

[6] SOCP : http://www.stanford.edu/˜boyd/SOCP/

[7] SP : http://www.ee.ucla.edu/˜vandenbe/sp/

[8] SDPpack : http://cs.nyu.edu/cs/faculty/overton/sdppack/sdppack.html

[9] SBMethod : http://www.zib.de/helmberg/SBmethod

[10] LOQO : http://www.princeton.edu/˜rvdb/

[11] MOSEK : http://www.mosek.com

[12] DSPD3 : http://www.mcs.anl.gov/˜benson/

[13] LMISOL : http://www.dt.fee.unicamp.br/˜mauricio/lmisol10.html

[14] SDPHA : http://www.math.uiowa.edu/˜rsheng/SDPHA/sdpha.html

[15] YALMIP : http://www.control.isy.liu.se/˜johanl/yalmip.html

[16] SDPSOL : http://www.stanford.edu/˜boyd/SDPSOL.html

[17] LMITOOL : http://ensta.fr/uer/uma/gropco/lmi

[18] LMITRANS : http://vehicle.me.berkeley.edu/˜guiness/lmitrans.html

[19] SeDuMiInterface : http://laas.fr/˜peaucelle

[20] F. Alizadeh, “Interior point methods in semidefinite programming with applications to combinatorial optimization”, SIAM J. Optimization, Vol, 5, pp13-51, 1995.

[21] C. Beck, “Computational issues in solving LMIs”, Proceedings of the CDC, Brighton, 1991.

[22] R. Bellman, K. Fan, “On systems of linear inequalities in Hermitian matrix variables”, Proc. Sympos. Pure Math., V.L. Klee, eds. pp. 1-11, 1963.

[23] A. Ben-Tal, A. Nemirovskii, “Convex optimization in engineering”, Technical Report, Technion University, 2000.

[24] S. Boyd, L. El Ghaoui, E. Feron, V. Balakrishnan, Linear matrix inequalities in systems and control theory, Vol. 15 of SIAM studies in Applied Mathematics, 1994.

[25] S. Boyd, “Convex optimization”, course reader EE236B, University of California, Los Angeles, 1997.

[26] J. Cullum, W.E. Donath, P. Wolfe, “The minimization of certain nondifferentiable sums of eigenvalues of symmetric matrices”, Mathematical Programming Study, 3, pp.

35-55, 1975.

[27] L. El Ghaoui, S. Niculescu, Advances in linear matrix inequality methods in control, SIAM advances in design and control, 2000.

[28] A.V. Fiacco, G.P. McCormick, Nonlinear programming: sequential unconstrained minimization techniques, John Wiley and Sons, NY, 1968.

[29] P. Gahinet, A. Nemirovskii, “The projective method for solving linear matrix inequalities”, Mathematical programming, Vol. 77, pp. 163-190, 1997.

[30] C. Helmberg, “Semidefinite programming for combinatorial optimization”, ZIB-Report 00-34, 2000.

[31] J.B. Iriart-Urruty, C. Lemarechal, Convex analysis and minimization algorithms I et II, Vol. 306, Springer-verlag, 1993.

[32] N. Karmakar, “A new polynomial-time algorithm for linear programming”, Combinatorica, Vol. 4, pp.373-395, 1984.

[33] M.S. Lobo, L. Vandenberghe, S. Boyd, H. Lebret, “Applications of second-order cone programming”, Linear algebra and its applications, Vol. 284, pp. 193-228, 1998.

[34] H.D. Mittelmann, “An independant benchmarking of SDP and SOCP solvers”, Septi eme DIMACS Implementation Challenge on Semidefinite and Related Optimization

Problems, Technical Report, Dept. of Mathematics, Arizona State University, 2001.

[35] Y. Nesterov, A. Nemirovskii, “A general approach to polynomial-time algorithms design for convex programming”, Technical Report, Centr. Econ. and Math. Inst., USSR

Acad. Sci., Moscow, USSR, 1988.

[36] Y. Nesterov, A. Nemirovskii, Interior-point polynomial algorithms in convex programming, Vol. 13 of SIAM studies in Applied Mathematics, 1994.

[37] B.T. Polyak, “Subgradient mehods: a survey of soviet research”, in Nonsmooth optimization, C. Lemar echal and R. Miffin, eds., Pergamon Press, NY, 1977.

[38] N.Z. Shor, Minimization methods for non-differentiable functions, Springer verlag, 1985.

[39] M.V. Ramana, “An exact duality theory for semidefinite programming and its complexity implications”, Mathematical programming, Vol. 77, pp. 129-162,1997.

[40] L. Vandenberghe, V. Balakrishnan, “Algorithms and software tools for LMI problems in control: an overview”, Proceedings of the symposium on computer-aided controlsystem design, Dearborn, 1996.

[41] L. Vandenberghe, S. Boyd, “Semidefinite programming”, SIAM Review, Vol. 38, pp. 49-95, 1996.

[42] L. Vandenberghe, S. Boyd, S. P. Wu, “Determinant maximization with linear matrix inequality constraints”, SIAM J. on Matrix Analysis and Applications, Vol. 19, No. 2,

pp. 499-533, 1998.

[43] H. Wolkowicz, R. Saigal, L. Vandenberghe, Handbook of semidefinite programming, Kluwer’s international series, 2000.

[44] S. Zhang, “Conic optimization and the duality theory”, HPOPT, Rotterdam, 1998.