Conic Optimisation:

Why, When and How Should it be Used.

A Short Trip Along a Basic yet Powerful Optimisation Tool

OSE Workshop 2015 – 17 September 2015 Glasgow (UK)

Andrea Cassioli, PhDandrea.cassioli@mosek.com

www.mosek.com

Conic Optimization I

Standard form

min∑k

cTk xk +∑k

〈Ck , Xk〉

s.t. ∑k

aTikxk +∑k

〈Aik , Xik〉 = bi i = 1, . . .

xk ∈ Kk k = 1, . . .Xk ∈ Kk k = 1, . . .

where Kk , Kk are proper cones.

1 / 24

Conic Optimization II

We only consider

Positive Orth. R+ = x ∈ Rn|x ≥ 0

Lorentz Q = (t, x) ∈ R× Rn|t ≥ ‖x‖2

Rotated Qr =

(z , y , x) ∈ R× R× Rn|2zy ≥ ‖x‖22, z ,w ≥ 0

PSD Matrices S+ =X ∈ Rn×n|X is PSD

Other more exotic cones can be defined, still an active researchfield.

2 / 24

Conic Optimization III

Properties

• convexity guarantee,

• rich duality theory,

• (almost) avoid ill-posedness,

• polynomial time complexity,

• deeply investigated, but still active research field,

• many good implementations!

3 / 24

Convexity guarantee

If you can cast your problem in conic form, then it is convex!No need for long and tedious convexity proof!

Plenty of rules and examples:

Boyd, S. and Vandenberghe, L., Convex optimization,Cambridge University Press, 2004.

Ben-Tal, A. and Nemirovski, A, Lectures on Modern ConvexOptimization: Analysis, Algorithms, and EngineeringApplications, SIAM, 2001.

Alizadeh, F. and Goldfarb, D. Second-order coneprogramming, Math. Programming 95-1, 2003.

4 / 24

Conic Modeling I

What can we represent with CO?

Trivially:

• LP

• QP

• QCQP

• SOCP

• SDP

5 / 24

Conic Modeling II

• Geometric mean:(x , t) ∈ Rn+1|(x1x2 . . . xn)1/n ≥ t, x ≥ 0

• Harmonic mean:

(x , t) ∈ Rn+1|(

1

2

∑x−1i

)−1≥ t, x ≥ 0

6 / 24

Conic Modeling III

• Increasing powers:(x , t)|xp/q ≤ t, x ≥ 0

, p/q ≥ 1, p, q ∈ Z+

• Decrease powers:(x , t)|x−p/q ≤ t, x ≥ 0

, p, q ∈ Z+

• Power cones:(x , y) ∈ Rn

+ × R| |y | ≤∏

xαj

j

, α > 0,

∑αj = 0

7 / 24

Conic Modeling IV

Eigenvalue related problems:

• largest/smallest eigenvalue

• eigenvalue spread

• spectral radius

• minimize condition number of a PSD matrix

• singular values.

8 / 24

Conic Modeling - Example

(Linear) Regressions/Regularization

min φ(δ)δ = Ax − yx ∈ Ω

φ(·) is typically

• ‖ · ‖1, ‖ · ‖2, ‖ · ‖∞• 3/2−regression

• k−largest penalty

• Huber penalty

• Dead-zone linear penalty

9 / 24

Conic Modeling - Example

Nearest correlation matrix

min ‖X − Y ‖FXii = 1 i = 1, . . . , nX ∈ Sn+

where ‖A‖F =√

tr(ATA) =

√∑i ,j

A2ij , and hence

min t(t, vec(X − Y )) ∈ QXii = 1 i = 1, . . . , nX ∈ Sn+

10 / 24

Conic Modeling - Examples

For a polyhedron, find an ellipsoid E defined by a matrix C :

Maximal inscribed ellipsoid: Vol(E) ≈ det(C )1/n

Minimal enclosing ellipsoid: Vol(E) ≈ det(C )−1/n

Much interest in the robotic community.11 / 24

Conic Modeling - Examples

Polynomial curve-fitting

For a given degree, condition on

• interpolation

• non-negativity

• monotonicity

• convexity/concavity

f2(t)

f4(t)

f8(t)12

1

32

−2 −1 0 1 2 t

12 / 24

Conic Modeling

There is plenty of material out there.

Advice

Always try to cast your problem in conic form!

13 / 24

CO polynomial complexity

The problem size is not just the combination of the number ofvariables and constraints, but also the number of bits to representthe problem!

Advise

Do not be afraid of transformations that introduce additionalvariables/constraints.

A larger but sparser problem can have the same complexity that asmaller but denser.

14 / 24

SOCP vs. QCQP

SOCP contains QCQP, which one should one use?

• in general no significant differences in complexity

• SOCP more general, ready for extensions

• some specialized algorithms for structured QP/QCQP exist

• our experience indicates SOCP more robust.

15 / 24

From QCQP to SOCP I

To convert QCQP to SOCP you need to factorize the Q matrix...

min xTQxx ∈ X

↓

min t

t ≥ xTQx = xTFFT x = ‖Fx‖2x ∈ X

↓

min t(1/2, t,Fx) ∈ Qr

x ∈ X16 / 24

From QCQP to SOCP II

Forming F may sound tricky, but it can pay:

• You can exploit the structure better than the solver;

• Sometimes F already given;

• You may need to factorize to check Q ∈ S+;

• Recover from errors.

17 / 24

From QCQP to SOCP III

minimize −x1 − x2subject to (x1 − x2)2 ≤ 0,

0 ≤ x1, x2 ≤ 1

What if the quadratic constraints is affected by a small error ε, i.e.

xT[

1 −1−1 1 + ε

]x ≤ 0

Observe:

• ε < 0 : The problem is not convex.

• ε = 0 : x∗1 = x∗2 = 1.

• ε > 0 : x∗1 = x∗2 = 0.

18 / 24

From QCQP to SOCP III

Advise

Do try to cast a QCQP as a SOCP yourself!

19 / 24

Exploit duality I

In CO duality allows to

1 Automatic dual formulation (pretty much like LPs)

2 Numeric infeasibility certificate (approximate Farkas’ Lemma)

For primal-dual algorithms there is no theoretical difference inusing primal or dual formulation, but...

20 / 24

Exploit duality II

in practice:

• presolver may be affected

• caching issue from different memory setup

• linear algebra routines may be tuned better for (dual) primalformulation.

21 / 24

Exploit duality

In our experience some times up to 10x speed up!

Advise

Give the dual formulation a chance!

Some framework like CVX will do it automagically for you...

22 / 24

Software

Great tools/solvers for CO available:

• CVX, YALMIP, Julia Convex, PICOS, ...

• MOSEK, CPLEX, Gurobi,...

• Sedumi, SDPA, SDPT3, CVXOPT, ...

Advice

Do not re-invent the wheel!

23 / 24

Conclusion

My suggestions:

1 use CO whenever possible

2 rely on proven technology even for cutting edge research

3 give the dual problem a chance

4 move to SOCP to QP/QCQP unless a special case is at hand

5 don’t be afraid of larger but sparser formulations

6 share your experiences and issues with the developers!

24 / 24

Thank you!

Andrea Cassioli, PhDandrea.cassioli@mosek.com

www.mosek.com