Convex Optimization Problems
Prof. Daniel P. PalomarThe Hong Kong University of Science and Technology (HKUST)
MAFS5310 - Portfolio Optimization with RMSc in Financial Mathematics
Fall 2020-21, HKUST, Hong Kong
Outline
1 Optimization Problems
2 Convex Optimization
3 Quasi-Convex Optimization
4 Classes of Convex Problems: LP, QP, SOCP, SDP
5 Multicriterion Optimization (Pareto Optimality)
6 Solvers
Outline
1 Optimization Problems
2 Convex Optimization
3 Quasi-Convex Optimization
4 Classes of Convex Problems: LP, QP, SOCP, SDP
5 Multicriterion Optimization (Pareto Optimality)
6 Solvers
Optimization Problem in Standard Form
General optimization problem in standard form:
minimizex
f0 (x)
subject to fi (x) ≤ 0 i = 1, . . . ,mhi (x) = 0 i = 1, . . . , p
wherex = (x1, . . . , xn) is the optimization variablef0 : Rn −→ R is the objective functionfi : Rn −→ R, i = 1, . . . ,m are inequality constraintfunctionshi : Rn −→ R, i = 1, . . . , p are equality constraintfunctions.
Goal: find an optimal solution x? that minimizes f0 while satisfying allthe constraints.
D. Palomar Convex Problems 4 / 48
Optimization Problem in Standard Form
Feasibility:a point x ∈ dom f0 is feasible if it satisfies all the constraints andinfeasible otherwisea problem is feasible if it has at least one feasible point and infeasibleotherwise.
Optimal value:
p? = inf {f0 (x) | fi (x) ≤ 0, i = 1, . . . ,m, hi (x) = 0, i = 1, . . . , p}
p? =∞ if problem infeasible (no x satisfies the constraints)p? = −∞ if problem unbounded below.
Optimal solution: x? such that f (x?) = p? (and x? feasible).
D. Palomar Convex Problems 5 / 48
Global and Local Optimality
A feasible x is optimal if f0 (x) = p?; Xopt is the set of optimal points.A feasible x is locally optimal if is optimal within a ball
Examples:f0 (x) = 1/x , dom f0 = R++: p? = 0, no optimal pointf0 (x) = x3 − 3x : p? = −∞, local optimum at x = 1.
D. Palomar Convex Problems 6 / 48
Implicit Constraints
The standard form optimization problem has an explicit constraint:
x ∈ D =m⋂i=0
dom fi ∩p⋂
i=1
dom hi
D is the domain of the problemThe constraints fi (x) ≤ 0, hi (x) = 0 are the explicit constraintsA problem is unconstrained if it has no explicit constraintsExample:
minimizex
log(b − aT x
)is an unconstrained problem with implicit constraint b > aT x .
D. Palomar Convex Problems 7 / 48
Feasibility Problem
Sometimes, we don’t really want to minimize any objective, just tofind a feasible point:
findx
x
subject to fi (x) ≤ 0 i = 1, . . . ,mhi (x) = 0 i = 1, . . . , p
This feasibility problem can be considered as a special case of ageneral problem:
minimizex
0
subject to fi (x) ≤ 0 i = 1, . . . ,mhi (x) = 0 i = 1, . . . , p
where p? = 0 if constraints are feasible and p? =∞ otherwise.
D. Palomar Convex Problems 8 / 48
Outline
1 Optimization Problems
2 Convex Optimization
3 Quasi-Convex Optimization
4 Classes of Convex Problems: LP, QP, SOCP, SDP
5 Multicriterion Optimization (Pareto Optimality)
6 Solvers
Convex Optimization Problem
Convex optimization problem in standard form:
minimizex
f0 (x)
subject to fi (x) ≤ 0 i = 1, . . . ,mAx = b
where f0, f1, . . . , fm are convex and equality constraints are affine.Local and global optima: any locally optimal point of a convexproblem is globally optimal.Most problems are not convex when formulated.Reformulating a problem in convex form is an art, there is nosystematic way.
D. Palomar Convex Problems 10 / 48
Example
The following problem is nonconvex (why not?):
minimizex
x21 + x2
2
subject to x1/(1 + x2
2)≤ 0
(x1 + x2)2 = 0
The objective is convex.The equality constraint function is not affine; however, we can rewriteit as x1 = −x2 which is then a linear equality constraint.The inequality constraint function is not convex; however, we canrewrite it as x1 ≤ 0 which again is linear.We can rewrite it as
minimizex
x21 + x2
2
subject to x1 ≤ 0x1 = −x2
D. Palomar Convex Problems 11 / 48
Equivalent Reformulations
Eliminating/introducing equality constraints:
minimizex
f0 (x)
subject to fi (x) ≤ 0 i = 1, . . . ,mAx = b
is equivalent to
minimizez
f0 (Fz + x0)
subject to fi (Fz + x0) ≤ 0 i = 1, . . . ,m
where F and x0 are such that Ax = b ⇐⇒ x = Fz + x0 for some z .
D. Palomar Convex Problems 12 / 48
Equivalent Reformulations
Introducing slack variables for linear inequalities:
minimizex
f0 (x)
subject to aTi x ≤ bi i = 1, . . . ,m
is equivalent to
minimizex ,s
f0 (x)
subject to aTi x + si = bi i = 1, . . . ,msi ≥ 0
D. Palomar Convex Problems 13 / 48
Equivalent Reformulations
Epigraph form: a standard form convex problem is equivalent to
minimizex ,t
t
subject to f0 (x)− t ≤ 0fi (x) ≤ 0 i = 1, . . . ,mAx = b
Minimizing over some variables:
minimizex ,y
f0 (x , y)
subject to fi (x) ≤ 0 i = 1, . . . ,m
is equivalent to
minimizex
f̃0 (x)
subject to fi (x) ≤ 0 i = 1, . . . ,m
where f̃0 (x) = infy f0 (x , y).D. Palomar Convex Problems 14 / 48
Outline
1 Optimization Problems
2 Convex Optimization
3 Quasi-Convex Optimization
4 Classes of Convex Problems: LP, QP, SOCP, SDP
5 Multicriterion Optimization (Pareto Optimality)
6 Solvers
Quasiconvex Optimization
Quasi-convex optimization problem in standard form:
minimizex
f0 (x)
subject to fi (x) ≤ 0 i = 1, . . . ,mAx = b
where f0 : Rn −→ R is quasiconvex and f1, . . . , fm are convex.Observe that it can have locally optimal points that are not (globally)optimal:
D. Palomar Convex Problems 16 / 48
Quasiconvex Optimization
Convex representation of sublevel sets of a quasiconvex function f0:there exists a family of convex functions φt (x) for fixed t such that
f0 (x) ≤ t ⇐⇒ φt (x) ≤ 0.
Example:
f0 (x) =p (x)
q (x)
with p convex, q concave, and p (x) ≥ 0, q (x) > 0 on dom f0. Wecan choose:
φt (x) = p (x)− tq (x)
for t ≥ 0, φt (x) is convex in xp (x) /q (x) ≤ t if and only if φt (x) ≤ 0.
D. Palomar Convex Problems 17 / 48
Quasiconvex Optimization
Solving a quasiconvex problem via convex feasibility problems: theidea is to solve the epigraph form of the problem with a sandwichtechnique in t:
for fixed t the epigraph form of the original problem reduces to afeasibility convex problem
φt (x) ≤ 0, fi (x) ≤ 0 ∀i , Ax ≤ b
if t is too small, the feasibility problem will be infeasibleif t is too large, the feasibility problem will be feasiblestart with upper and lower bounds on t (termed u and l , resp.) anduse a sandwitch technique (bisection method): at each iteration uset = (l + u) /2 and update the bounds according to thefeasibility/infeasibility of the problem.
D. Palomar Convex Problems 18 / 48
Outline
1 Optimization Problems
2 Convex Optimization
3 Quasi-Convex Optimization
4 Classes of Convex Problems: LP, QP, SOCP, SDP
5 Multicriterion Optimization (Pareto Optimality)
6 Solvers
Linear Programming (LP)
LP:minimize
xcT x + d
subject to Gx ≤ hAx = b
Convex problem: affine objective and constraint functions.Feasible set is a polyhedron:
D. Palomar Convex Problems 20 / 48
`1- and `∞- Norm Problems as LPs
`∞-norm minimization:
minimizex
‖x‖∞subject to Gx ≤ h
Ax = b
is equivalent to the LP
minimizet,x
t
subject to −t1 ≤ x ≤ t1Gx ≤ hAx = b.
D. Palomar Convex Problems 21 / 48
`1- and `∞- Norm Problems as LPs
`1-norm minimization:
minimizex
‖x‖1subject to Gx ≤ h
Ax = b
is equivalent to the LP
minimizet,x
∑i ti
subject to −t ≤ x ≤ tGx ≤ hAx = b.
D. Palomar Convex Problems 22 / 48
Linear-Fractional Programming
Linear-fractional programming:
minimizex
(cT x + d
)/(eT x + f
)subject to Gx ≤ h
Ax = b
with dom f0 ={x | eT x + f > 0
}.
It is a quasiconvex optimization problem (solved by bisection).Interestingly, the following LP is equivalent:
minimizey ,z
cT y + dz
subject to Gy ≤ hzAy = bzeT y + fz = 1z ≥ 0
D. Palomar Convex Problems 23 / 48
Quadratic Programming (QP)
Quadratic programming:
minimizex
(1/2) xTPx + qT x + r
subject to Gx ≤ hAx = b
Convex problem (assuming P ∈ Sn � 0): convex quadratic objectiveand affine constraint functions.Minimization of a convex quadratic function over a polyhedron:
D. Palomar Convex Problems 24 / 48
Quadratically Constrained QP (QCQP)
Quadratically constrained QP:
minimizex
(1/2) xTP0x + qT0 x + r0
subject to (1/2) xTPix + qTi x + ri ≤ 0 i = 1, . . . ,mAx = b
Convex problem (assuming Pi ∈ Sn � 0): convex quadratic objectiveand constraint functions.
D. Palomar Convex Problems 25 / 48
Second-Order Cone Programming (SOCP)
Second-order cone programming:
minimizex
f T x
subject to ‖Aix + bi‖ ≤ cTi x + di i = 1, . . . ,mFx = g
Convex problem: linear objective and second-order cone constraintsFor Ai row vector, it reduces to an LP.For ci = 0, it reduces to a QCQP.More general than QCQP and LP.
D. Palomar Convex Problems 26 / 48
Semidefinite Programming (SDP)
SDP:minimize
xcT x
subject to x1F1 + x2F2 + · · ·+ xnFn � GAx = b
Inequality constraint is called linear matrix inequality (LMI).Convex problem: linear objective and linear matrix inequality (LMI)constraints.Observe that multiple LMI constraints can always be written as asingle one.
D. Palomar Convex Problems 27 / 48
Semidefinite Programming (SDP)
LP and equivalent SDP:
minimizex
cT x
subject to Ax ≤ bminimize
xcT x
subject to diag (Ax − b) � 0SOCP and equivalent SDP:
minimizex
f T x
subject to ‖Aix + bi‖ ≤ cTi x + di , i = 1, . . . ,m
minimizex
f T x
subject to[ (
cTi x + di)I Aix + bi
(Aix + bi )T cTi x + di
]� 0, i = 1, . . . ,m
D. Palomar Convex Problems 28 / 48
Semidefinite Programming (SDP)
Eigenvalue minimization:
minimizex
λmax (A (x))
where A (x) = A0 + x1A1 + · · ·+ xnAn, is equivalent to SDP
minimizex
t
subject to A (x) � tI
It follows from
λmax (A (x)) ≤ t ⇐⇒ A (x) � tI
D. Palomar Convex Problems 29 / 48
Outline
1 Optimization Problems
2 Convex Optimization
3 Quasi-Convex Optimization
4 Classes of Convex Problems: LP, QP, SOCP, SDP
5 Multicriterion Optimization (Pareto Optimality)
6 Solvers
Scalarization for Multicriterion Problems
Consider a multicriterion optimization with q different objectives:
f0 (x) = (F1 (x) , . . . ,Fq (x)) .
To find Pareto optimal points, minimize the positive weighted sum:
λT f0 (x) = λ1F1 (x) + · · ·+ λqFq (x) .
Example: regularized least-squares:
minimizex
‖Ax − b‖22 + γ ‖x‖22 .
D. Palomar Convex Problems 31 / 48
References on Convex Problems
Chapter 4 ofStephen Boyd and Lieven Vandenberghe, Convex Optimization.Cambridge, U.K.: Cambridge University Press, 2004.
https://web.stanford.edu/~boyd/cvxbook/
D. Palomar Convex Problems 32 / 48
Outline
1 Optimization Problems
2 Convex Optimization
3 Quasi-Convex Optimization
4 Classes of Convex Problems: LP, QP, SOCP, SDP
5 Multicriterion Optimization (Pareto Optimality)
6 Solvers
Solvers
A solver is an engine for solving a particular type of mathematicalproblem, such as a convex program.Every programming language (e.g., Matlab, Octave, R, Python, C,C++) has a long list of available solvers to choose from.Solvers typically handle only a certain class of problems, such as LPs,QPs, SOCPs, SDPs, or GPs.They also require that problems be expressed in a standard form.Most problems do not immediately present themselves in a standardform, so they must be transformed into standard form.
D. Palomar Convex Problems 34 / 48
Solver Example: Matlab’s linprog
A program for solving LPs:
x = linprog( c, A, b, A_eq, B_eq, l, u )
Problems must be expressed in the following standard form:
minimizex
cT x
subject to Ax ≤ bAeqx = beql ≤ x ≤ u
D. Palomar Convex Problems 35 / 48
Conversion to Standard Form: Common Tricks
Representing free variables as the difference of nonnegative variables:
x free =⇒ x+ − x−, x+ ≥ 0, x− ≥ 0
Elimininating inequality constraints using slack variables:
aT x ≤ b =⇒ aT x + s = b, s ≥ 0
Splitting equality constraints into inequalities:
aT x = b =⇒ aT x ≤ b, aT x ≥ b
D. Palomar Convex Problems 36 / 48
Solver Example: SeDuMi
A program for solving LPs, SOCPs, SDPs, and related problems:
x = sedumi( A, b, c, K )
Solves problems of the form:
minimizex
cT x
subject to Ax = bx ∈ K , K1 ×K2 × · · · × KL
where each set Ki ⊆ Rni , i = 1, 2, . . . , L is chosen from a very shortlist of cones.The Matlab variable K gives the number, types, and dimensions of thecones Ki .
D. Palomar Convex Problems 37 / 48
Solver Example: SeDuMi
Cones supported by SeDuMi:free variables: Rni
a nonnegative orthant: Rni+ (for linear inequalities)
a real or complex second-order cone:
Qn , {(x, y) ∈ Rn × R | ‖x‖2 ≤ y}Qn
c , {(x, y) ∈ Cn × R | ‖x‖2 ≤ y}
a real or complex semidefinite cone:
Sn+ ,{X ∈ Rn×n | X = XT , X � 0
}Hn+ ,
{X ∈ Cn×n | X = XH , X � 0
}The cones must be arranged in this order, i.e., the free variables first, thenthe nonnegative orthants, then the second-order cones, then thesemidefinite cones.
D. Palomar Convex Problems 38 / 48
Example: Norm Approximation
Consider the norm approximation problem:
minimizex
‖Ax− b‖
An optimal value x? minimizes the residuals
rk = aTk x− bk , k = 1, 2, . . . ,m
according to the measure defined by the norm ‖·‖Obviously, the value of x? depends significantly upon the choice ofthat norm...... and so does the process of conversion to standard form
D. Palomar Convex Problems 39 / 48
Example: Euclidean or `2-Norm
Norm approximation problem:
minimizex
‖Ax− b‖2 =(∑m
k=1(aTk x− bk
)2)1/2
No need to use any solver here: this is a least squares (LS) problem,with an analytic solution:
x? = (ATA)−1ATb
In Matlab or Octave, a single command computes the solution:>> x = A \ b
Similarly, in R:>> x = solve(A, b)
D. Palomar Convex Problems 40 / 48
Example: Chebyshev or `∞-Norm
Norm approximation problem:
minimizex
‖Ax− b‖∞ = minimizex
max1≤k≤m
∣∣aTk x− bk∣∣
This can be expressed as a linear program:
minimizex,t
t
subject to −t1 ≤ Ax− b ≤ t1
or, equivalently,
minimizex,t
[0T 1
] [ xt
]subject to
[A −1−A −1
] [xt
]≤[
b−b
]
D. Palomar Convex Problems 41 / 48
Example: Chebyshev or `∞-Norm
Recall the final formulation:
minimizex,t
[0T 1
] [ xt
]subject to
[A −1−A −1
] [xt
]≤[
b−b
]
Matlab’s linprog call:>> xt = linprog( [zeros(n,1); 1], ...
[A,-ones(m,1); -A,-ones(m,1)], ...[b; -b] )
>> x = xt(1:n)
D. Palomar Convex Problems 42 / 48
Example: Manhattan or `1-Norm
Norm approximation problem:
minimizex
‖Ax− b‖1 = minimizex
∑mk=1
∣∣aTk x− bk∣∣
This can be expressed as a linear program:
minimizex,t
1T t
subject to −t ≤ Ax− b ≤ t
or, equivalently,
minimizex,t
[0T 1T
] [ xt
]subject to
[A −I−A −I
] [xt
]≤[
b−b
]
D. Palomar Convex Problems 43 / 48
Example: Manhattan or `1-Norm
Recall the final formulation:
minimizex,t
[0T 1T
] [ xt
]subject to
[A −I−A −I
] [xt
]≤[
b−b
]
Matlab’s linprog call:>> xt = linprog( [zeros(n,1); ones(n,1)], ...
[A,-eye(m,1); -A,-eye(m,1)], ...[b; -b] )
>> x = xt(1:n)
D. Palomar Convex Problems 44 / 48
Example: Constrained Euclidean or `2-Norm
Constrained norm approximation problem:
minimizex
‖Ax− b‖2subject to Cx = d
l ≤ x ≤ u
This is not a least squares problem, but it is QP and an SOCP.This can be expressed as
minimizex,y,t,sl ,su
t
subject to Ax− b = yCx = dx− sl = lx + su = usl , su ≥ 0‖y‖2 ≤ t
D. Palomar Convex Problems 45 / 48
Example: Constrained Euclidean or `2-Norm
Equivalently: minimizex,y,t,sl ,su
[0T 0T 0T 0T 1
]x̄
subject to
A −ICI −II I
x̄ ≤
bdlu
x̄ ∈ Rn × Rn
+ × Rn+ × Qm
SeDuMi call:>> AA = [ A, zeros(m,n), zeros(m,n), -eye(m), 0;
C, zeros(p,n), zeros(p,n) zeros(p,n), 0;eye(n), -eye(n), zeros(n,n), zeros(n,n), 0;eye(n), zeros(n,n), eye(n), zeros(n,n), 0 ]
>> bb = [ b; d; l; u ]>> cc = [ zeros(3*n+m,1); 1 ]>> K.f = n; K.l = 2*n; K.q = m + 1;>> xsyz = sedumi( AA, bb, cc, K )>> x = xsyz(1:n)D. Palomar Convex Problems 46 / 48
Modeling Frameworks: cvx
A modeling framework simplifies the use of a numerical technology byshielding the user from the underlying mathematical details.Examples: SDPSOL, YALMIP, cvx, etc.cvx is designed to support convex optimization or, more specifically,disciplined convex programming (available in Matlab, Python, R, Julia,Octave, etc.)People don’t simply write optimization problems and hope that theyare convex; instead they draw from a "mental library"of functions andsets with known convexity properties and combine them in ways thatconvex analysis guarantees will produce convex results.Disciplined convex programming formalizes this methodology.Links:
cvx: http://cvxr.comcvx user guide: http://web.cvxr.com/cvx/doc/CVX.pdfcvx for R (cvxr): https://github.com/cvxgrp/CVXR
D. Palomar Convex Problems 47 / 48
Thanks
For more information visit:
https://www.danielppalomar.com