Convex Functions (I)

Lijun Zhangzlj@nju.edu.cnhttp://cs.nju.edu.cn/zlj

Outline Basic Properties

Definition First-order Conditions, Second-order Conditions Jensen’s inequality and extensions Epigraph

Operations That Preserve Convexity Nonnegative Weighted Sums Composition with an affine mapping Pointwise maximum and supremum Composition Minimization Perspective of a function

Summary

Outline Basic Properties

Definition First-order Conditions, Second-order Conditions Jensen’s inequality and extensions Epigraph

Operations That Preserve Convexity Nonnegative Weighted Sums Composition with an affine mapping Pointwise maximum and supremum Composition Minimization Perspective of a function

Summary

Convex Function

is convex if is convex

Convex Function

is convex if is convex

is strictly convex if

Convex Function

is convex if is convex

is concave if is convex is convex

Affine functions are both convex and concave, and vice versa.

Extended-value Extensions

The extended-value extension of is

Example

Extended-value Extensions

The extended-value extension of is

Example Indicator Function of a Set

Zeroth-order Condition

Definition High-dimensional space

A function is convex if and only if it is convex when restricted to any line that intersects its domain. , is convex is convex One-dimensional space

First-order Conditions

is differentiable. Then is convex if and only if is convex For all

𝑓 𝑦 𝑓 𝑥 𝛻𝑓 𝑥 𝑦 𝑥

First-order Taylor approximation

First-order Conditions

is differentiable. Then is convex if and only if is convex For all

Local Information Global Information

is strictly convex if and only if

𝑓 𝑦 𝑓 𝑥 𝛻𝑓 𝑥 𝑦 𝑥

𝑓 𝑦 𝑓 𝑥 𝛻𝑓 𝑥 𝑦 𝑥

Proof

is convex

Necessary condition:𝑓 𝑥 𝑡 𝑦 𝑥 1 𝑡 𝑓 𝑥 𝑡𝑓 𝑦 , 0 𝑡 1

⇒ 𝑓 𝑦 𝑓 𝑥→

𝑓 𝑦 𝑓 𝑥 𝑓 𝑥 𝑦 𝑥

Sufficient condition:𝑧 𝜃𝑥 1 𝜃 𝑦

𝑓 𝑥 𝑓 𝑧 𝑓 𝑧 𝑥 𝑧𝑓 𝑦 𝑓 𝑧 𝑓 𝑧 𝑦 𝑧

⇒ 𝑓 𝑥 𝑓 𝑧 1 𝜃 𝑓 𝑧 𝑥 𝑦𝑓 𝑦 𝑓 𝑧 𝜃𝑓 𝑧 𝑥 𝑦

⇒ 𝜃𝑓 𝑥 1 𝜃 𝑓 𝑦 𝑓 𝑧 ⇒ 𝑓 𝜃𝑥 1 𝜃 𝑦 𝜃𝑓 𝑥 1 𝜃 𝑓 𝑦

Proof

is convex

is convex

𝑓 is convex ⇒ is convex ⇒ 𝑔 1 𝑔 0𝑔 0 ⇒ 𝑓 𝑦 𝑓 𝑥 𝛻𝑓 𝑥 𝑦 𝑥

𝑔 𝑡 𝑓 𝑡𝑦 1 𝑡 𝑥 , 𝑔′ 𝑡 𝛻𝑓 𝑡𝑦 1 𝑡 𝑥 𝑦 𝑥

𝑓 is convex

𝑔 is convex

First-order condition of 𝑔

First-order condition of 𝑓

Proof

is convex

is convex

⇒ 𝑔 𝑡 𝑔 𝑡 𝑔 𝑡 𝑡 𝑡 ⇒ 𝑔 𝑡 is convex ⇒𝑓 is convex

𝑔 𝑡 𝑓 𝑡𝑦 1 𝑡 𝑥 ,

𝑓 𝑡𝑦 1 𝑡 𝑥 𝑓 𝑡𝑦 1 𝑡 𝑥 𝛻𝑓 𝑡𝑦 1 𝑡 𝑥 𝑦 𝑥 𝑡 𝑡

𝑔′ 𝑡 𝛻𝑓 𝑡𝑦 1 𝑡 𝑥 𝑦 𝑥

𝑓 is convex

𝑔 is convex

First-order condition of 𝑔

First-order condition of 𝑓

Second-order Conditions

is twice differentiable. Then is convex if and only if is convex For all ,

Attention is strictly convex is strict convex

is strict convex but is convex is necessary,

Examples

Functions on is convex on ,

is convex on when or ,

and concave for

, for , is convex on

is concave on

Negative entropy is convex on

Examples

Functions on Every norm on is convex

Quadratic-over-linear: dom 𝑓 𝑥, 𝑦 ∈ 𝐑 𝑦 0

/ is concave on

is concave on

Examples

Functions on Every norm on is convex

𝑓 𝑥 is a norm on 𝐑

𝑓 𝜃𝑥 1 𝜃 𝑦 𝑓 𝜃𝑥 𝑓 1 𝜃 𝑦

𝜃𝑓 𝑥 1 𝜃 𝑓 𝑦

𝑓 𝜃𝑥 1 𝜃 𝑦 max 𝜃𝑥 1 𝜃 𝑦

𝜃max 𝑥 1 𝜃 max 𝑦

Examples

Functions on

𝛻 𝑓 𝑥, 𝑦 𝑦 𝑥𝑦𝑥𝑦 𝑥

𝑦𝑥 𝑦

𝑥 ≽ 0

Examples

Functions on

Examples

Functions on

𝛻 𝑓 𝑥𝟏

𝟏 𝑧 diag 𝑧 𝑧𝑧

𝑧 𝑒 , … 𝑒

𝑣 𝛻 𝑓 𝑥 𝑣𝟏

∑ 𝑧 ∑ 𝑣 𝑧

∑ 𝑣 𝑧 0

Cauchy-Schwarz inequality: 𝑎 𝑎 𝑏 𝑏

𝑎 𝑏

Examples

Functions on is concave on

𝑔 𝑡 𝑓 𝑍 𝑡𝑉 , 𝑍 𝑡𝑉 ≻ 0, 𝑍 ≻ 0

𝑔 𝑡 log det 𝑍 𝑡𝑉

log det 𝑍 𝐼 𝑡𝑍 𝑉𝑍 𝑍

∑ log 1 𝑡𝜆 log det 𝑍

𝜆 , … 𝜆 are the eigenvalues of 𝑍 𝑉𝑍

𝑔 𝑡 ∑ , 𝑔 𝑡 ∑

det 𝐴𝐵 det 𝐴 det 𝐵 https://en.wikipedia.org/wiki/Determinant

Sublevel Sets

-sublevel set

is convex is convex is convex is convex

-superlevel set

is concave convex

is convex

𝐶 𝑥 ∈ dom 𝑓 𝑓 𝑥 𝛼

𝐶 𝑥 ∈ dom 𝑓 𝑓 𝑥 𝛼

Epigraph

Graph of function

Epigraph of function

Epigraph

Epigraph of function

Hypograph

Conditions is convex is convex is concave is convex

Example

Matrix Fractional Function

Quadratic-over-linear:

Schur complement condition is convex Recall Example 2.10 in the book

𝑓 𝑥, 𝑌 𝑥 𝑌 𝑥, dom 𝑓 𝐑 𝐒

Application of Epigraph

First order Condition 𝑓 𝑦 𝑓 𝑥 𝛻𝑓 𝑥 𝑦 𝑥

𝑦, 𝑡 ∈ epi 𝑓 ⇒ 𝑡 𝑓 𝑦 𝑓 𝑥 𝛻𝑓 𝑥 𝑦 𝑥

Application of Epigraph

First order Condition 𝑓 𝑦 𝑓 𝑥 𝛻𝑓 𝑥 𝑦 𝑥

𝑦, 𝑡 ∈ epi 𝑓 ⇒ 𝑡 𝑓 𝑥 𝛻𝑓 𝑥 𝑦 𝑥

𝑦, 𝑡 ∈ epi 𝑓 ⇒ 𝛻𝑓 𝑥1

𝑦𝑡

𝑥𝑓 𝑥 0

Jensen’s Inequality

Basic inequality

points

Jensen’s Inequality

Infinite points

𝑓 𝑥 𝐄𝑓 𝑥 𝑧 , 𝑧 is a zero-mean noisy

Hölder’s inequality

Outline Basic Properties

Definition First-order Conditions, Second-order Conditions Jensen’s inequality and extensions Epigraph

Operations That Preserve Convexity Nonnegative Weighted Sums Composition with an affine mapping Pointwise maximum and supremum Composition Minimization Perspective of a function

Summary

Nonnegative Weighted Sums Finite sums 𝑤 0, 𝑓 is convex 𝑓 𝑤 𝑓 ⋯ 𝑤 𝑓 is convex

Infinite sums 𝑓 𝑥, 𝑦 is convex in 𝑥, ∀𝑦 ∈ 𝒜, 𝑤 𝑦 0 𝑔 𝑥 𝑓 𝑥, 𝑦 𝑤 𝑦𝒜 𝑑𝑦 is convex

Epigraph interpretation 𝐞𝐩𝐢 𝑤𝑓 𝑥, 𝑡 |𝑤𝑓 𝑥 𝑡

𝐼 00 𝑤 𝐞𝐩𝐢 𝑓 𝑥, 𝑤𝑡 |𝑓 𝑥 𝑡

𝐞𝐩𝐢 𝑤𝑓 𝐼 00 𝑤 𝐞𝐩𝐢 𝑓

Composition with an affine mapping

Affine Mapping

If is convex, so is

If is concave, so is

𝑔 𝑥 𝑓 𝐴𝑥 𝑏

Pointwise Maximum

is convex

is convex with

max 𝑓 𝜃𝑥 1 𝜃 𝑦 , 𝑓 𝜃𝑥 1 𝜃 𝑦max 𝜃𝑓 𝑥 1 𝜃 𝑓 𝑦 , 𝜃𝑓 𝑥 1

𝜃 𝑓 𝑦𝜃 max 𝑓 𝑥 , 𝑓 𝑥 1 𝜃 max 𝑓 𝑦 , 𝑓 𝑦𝜃𝑓 𝑥 1 𝜃 𝑓 𝑦

𝑓 , … 𝑓 is convex ⇒ 𝑓 𝑥 max 𝑓 𝑥 , … 𝑓 𝑥

Examples

Piecewise-linear functions

Sum of largest components

is convex

Pointwise maximum of !! !

linear functions

Pointwise Supremum

is convex in

is convex with

∈𝒜

Epigraph interpretation ∈𝒜

Intersection of convex sets is convex Pointwise infimum of a set of

concave functions is concave

∈𝒜

Examples

Support function of a set

∈

Distance to farthest point of a set

∈

Examples

Maximum eigenvalue of a symmetric matrix

Norm of a matrix is maximum singular value

of

Representation

Almost every convex function can be expressed as the pointwise supremum of a family of affine functions.

⟹ 𝑓 𝑥 sup 𝑔 𝑥 |𝑔 affine, 𝑔 z 𝑓 𝑧 ∀𝑧

𝑓: 𝐑 → 𝐑 is convex and dom 𝑓 𝐑

Compositions

Definition

Chain Rule

𝑓 𝑥 ℎ 𝑔 𝑥

𝛻 𝑓 𝑥 ℎ 𝑔 𝑥 𝛻 𝑔 𝑥 ℎ 𝑔 𝑥 𝛻𝑔 𝑥 𝛻𝑔 𝑥

Scalar Composition

and are twice differentiable

is convex, if

ℎ is convex and nondecreasing 𝑔 is convex

ℎ is convex and nonincreasing, 𝑔 is concave

𝑓 𝑥 ℎ 𝑔 𝑥 𝑔′ 𝑥 ℎ 𝑔 𝑥 𝑔′′ 𝑥

Scalar Composition

and are twice differentiable

is concave, if

ℎ is concave and nondecreasing 𝑔 is concave

ℎ is concave and nonincreasing, 𝑔 is convex

𝑓 𝑥 ℎ 𝑔 𝑥 𝑔′ 𝑥 ℎ 𝑔 𝑥 𝑔′′ 𝑥

Without differentiability assumption Without domain condition

ℎ 𝑥 0 with dom ℎ 1,2 , which is convex and non-decreasing

𝑔 𝑥 𝑥 with dom 𝑔 𝐑, which is convex

dom 𝑓 2, 1 ∪ 1, 2

𝑓 𝑥 ℎ 𝑔 𝑥 0

Scalar Composition

Without differentiability assumption Without domain condition

ℎ is convex, ℎ is nondecreasing, and 𝑔 is convex ⇒ 𝑓 is convex

ℎ is convex, ℎ is nonincreasing, and 𝑔 is concave ⇒ 𝑓 is convex

The conditions for concave are similar

Scalar Composition

Extended-value Extensions

Examples is convex is convex is concave and positive is

concave is concave and positive is

convex is convex and nonnegative and

is convex is convex is convex on

Vector Composition

ℎ and 𝑔 are twice differentiable dom 𝑔 𝐑, dom ℎ 𝐑

𝑓 ℎ ∘ 𝑔 ℎ 𝑔 𝑥 , … , 𝑔 𝑥

𝑓 𝑥 𝑔 𝑥 𝛻 ℎ 𝑔 𝑥 𝑔′ 𝑥 𝛻ℎ 𝑔 𝑥 𝑔′′ 𝑥

𝑓′ 𝑥 𝛻ℎ 𝑔 𝑥 𝑔′ 𝑥

Vector Composition

ℎ and 𝑔 are twice differentiable dom 𝑔 𝐑, dom ℎ 𝐑

𝑓 is convex, if 𝑓 𝑥 0 ℎ is convex, ℎ is nondecreasing in each

argument, and 𝑔 are convex ℎ is convex, ℎ is nonincreasing in each

argument, and 𝑔 are concave

𝑓 ℎ ∘ 𝑔 ℎ 𝑔 𝑥 , … , 𝑔 𝑥

𝑓 𝑥 𝑔 𝑥 𝛻 ℎ 𝑔 𝑥 𝑔′ 𝑥 𝛻ℎ 𝑔 𝑥 𝑔′′ 𝑥

Vector Composition

ℎ and 𝑔 are twice differentiable dom 𝑔 𝐑, dom ℎ 𝐑

𝑓 is concave, if 𝑓 𝑥 0 ℎ is concave, ℎ is nondecreasing in each

argument, and 𝑔 are concave

The general case is similar

𝑓 ℎ ∘ 𝑔 ℎ 𝑔 𝑥 , … , 𝑔 𝑥

𝑓 𝑥 𝑔 𝑥 𝛻 ℎ 𝑔 𝑥 𝑔′ 𝑥 𝛻ℎ 𝑔 𝑥 𝑔′′ 𝑥

Examples ℎ 𝑧 𝑧 ⋯ 𝑧 , 𝑧 ∈ 𝐑 , 𝑔 , … , 𝑔 is convex

⇒ ℎ ∘ 𝑔 is convex

ℎ 𝑧 log ∑ 𝑒 , 𝑔 , … , 𝑔 is convex ⇒ ℎ ∘𝑔 is convex

ℎ 𝑧 ∑ 𝑧/ on 𝐑 is concave for 0 𝑝

1, and its extension is nondecreasing. If 𝑔 is concave and nonnegative ⇒ ℎ ∘ 𝑔 is concave

Minimization is convex in is convex 𝑔 𝑥 inf

∈𝑓 𝑥, 𝑦 is convex if 𝑔 𝑥

∞, ∀ 𝑥 ∈ dom 𝑔 dom 𝑔 𝑥 𝑥, 𝑦 ∈ dom 𝑓 for some 𝑦 ∈ 𝐶

Proof by Epigraph epi 𝑔 𝑥, 𝑡 | 𝑥, 𝑦, 𝑡 ∈ epi 𝑓 for some 𝑦 ∈ 𝐶 The projection of a convex set is convex.

Examples Schur complement 𝑓 𝑥, 𝑦 𝑥 𝐴𝑥 2𝑥 𝐵𝑦 𝑦 𝐶𝑦 𝐴 𝐵

𝐵 𝐶 ≽ 0, 𝐴, 𝐶 is symmetric ⇒ 𝑓 𝑥, 𝑦 is convex 𝑔 𝑥 inf 𝑓 𝑥, 𝑦 𝑥 𝐴 𝐵𝐶 𝐵 𝑥 is convex

⇒ 𝐴 𝐵𝐶 𝐵 ≽ 0, 𝐶 is the pseudo-inverse of 𝐶

Distance to a set 𝑆 is a convex nonempty set,𝑓 𝑥, 𝑦 ‖𝑥 𝑦‖ is

convex in 𝑥, 𝑦 𝑔 𝑥 dist 𝑥, 𝑆 inf

∈‖𝑥 𝑦‖

Examples

Affine domain ℎ 𝑦 is convex 𝑔 𝑥 inf ℎ 𝑦 |𝐴𝑦 𝑥 is convex

Proof

𝑓 𝑥, 𝑦 ℎ 𝑦 if 𝐴𝑦 𝑥 ∞ otherwise

𝑓 𝑥, 𝑦 is convex in 𝑥, 𝑦 𝑔 is the minimum of 𝑓 over 𝑦

Perspective of a function

defined as

is the perspective of dom 𝑔 𝑥, 𝑡 |𝑥/𝑡 ∈ dom 𝑓, 𝑡 0 𝑓 is convex ⇒ 𝑔 is convex

Proof

Perspective mapping preserve convexity

𝑥, 𝑡, 𝑠 ∈ epi 𝑔 ⇔ 𝑡𝑓𝑥𝑡 𝑠

⇔ 𝑓𝑥𝑡

𝑠𝑡

⇔ 𝑥/𝑡, 𝑠/𝑡 ∈ epi 𝑓

Example

Euclidean norm squared

Composition with an Affine function is convex

is convex

Outline Basic Properties

Definition First-order Conditions, Second-order Conditions Jensen’s inequality and extensions Epigraph

Operations That Preserve Convexity Nonnegative Weighted Sums Composition with an affine mapping Pointwise maximum and supremum Composition Minimization Perspective of a function

Summary

Summary Basic Properties

Definition First-order Conditions, Second-order Conditions Jensen’s inequality and extensions Epigraph

Operations That Preserve Convexity Nonnegative Weighted Sums Composition with an affine mapping Pointwise maximum and supremum Composition Minimization Perspective of a function