An Analysis of Convex Relaxations M. Pawan Kumar Vladimir Kolmogorov Philip Torr for MAP Estimation.

transcript

An Analysis of Convex Relaxations

M. Pawan Kumar

Vladimir Kolmogorov

Philip Torr

for MAP Estimation

V1 V2 V3 V4

Label ‘0’

Label ‘1’

Labelling m = {1, 0, 0, 1}

Random Variables V = {V1, ... ,V4}

Label Set L = {0, 1}

• To analyze convex relaxations for MAP estimation

V1 V2 V3 V4

Label ‘0’

Label ‘1’

Cost(m) = 2 + 1 + 2 + 1 + 3 + 1 + 3 = 13

Which approximate algorithm is the best?

Minimum Cost Labelling? NP-hard problem

V1 V2 V3 V4

Label ‘0’

Label ‘1’

Objectives• Compare existing convex relaxations – LP, QP and SOCP

• Develop new relaxations based on the comparison

Outline

• Integer Programming Formulation

• Existing Relaxations

• Comparison

• Generalization of Results

• Two New SOCP Relaxations

Integer Programming Formulation

Label ‘0’

Label ‘1’Unary Cost

Unary Cost Vector u = [ 5

Cost of V1 = 0

Cost of V1 = 1

; 2 4 ]

Labelling m = {1 , 0}

Label ‘0’

Unary Cost Vector u = [ 5 2 ; 2 4 ]T

Label vector x = [ -1

V1 = 1

; 1 -1 ]T

Recall that the aim is to find the optimal x

Label ‘0’

Unary Cost Vector u = [ 5 2 ; 2 4 ]T

Label vector x = [ -1 1 ; 1 -1 ]T

Sum of Unary Costs = 12

∑i ui (1 + xi)

Label ‘0’

Label ‘1’Pairwise Cost

0Cost of V1 = 0 and V1 = 0

0Cost of V1 = 0 and V2 = 0

Cost of V1 = 0 and V2 = 11 0

Pairwise Cost Matrix P

Label ‘0’

Sum of Pairwise Costs14

∑ij Pij (1 + xi)(1+xj)

Label ‘0’

Sum of Pairwise Costs14

∑ij Pij (1 + xi +xj + xixj)

∑ij Pij (1 + xi + xj + Xij)=

X = x xT Xij = xi xj

Constraints

• Uniqueness Constraint

∑ xi = 2 - |L|i Va

• Integer Constraints

xi {-1,1}

X = x xT

x* = argmin 12

∑ ui (1 + xi) + 14

∑ Pij (1 + xi + xj + Xij)

∑ xi = 2 - |L|i Va

xi {-1,1}

X = x xT

ConvexNon-Convex

Outline

• Existing Relaxations– Linear Programming (LP-S)– Semidefinite Programming (SDP-L)– Second Order Cone Programming (SOCP-MS)

• Comparison

x* = argmin 12

∑ ui (1 + xi) + 14

∑ xi = 2 - |L|i Va

xi {-1,1}

X = x xT

Retain Convex PartSchlesinger, 1976

Relax Non-ConvexConstraint

x* = argmin 12

∑ ui (1 + xi) + 14

∑ xi = 2 - |L|i Va

xi [-1,1]

X = x xT

x* = argmin 12

∑ ui (1 + xi) + 14

∑ xi = 2 - |L|i Va

Xij [-1,1] 1 + xi + xj + Xij ≥ 0

∑ Xij = (2 - |L|) xij Vb

LP-Sxi [-1,1]

Outline• Integer Programming Formulation

• Comparison

• Experiments

x* = argmin 12

∑ ui (1 + xi) + 14

∑ xi = 2 - |L|i Va

xi {-1,1}

X = x xT

Retain Convex PartLasserre, 2000

x* = argmin 12

∑ ui (1 + xi) + 14

∑ xi = 2 - |L|i Va

xi [-1,1]

X = x xT

Retain Convex Part

Lasserre, 2000

x* = argmin 12

∑ ui (1 + xi) + 14

∑ xi = 2 - |L|i Va

xi [-1,1]

Retain Convex Part

Xii = 1 X - xxT 0

Accurate Inefficient

Lasserre, 2000

Outline

• Comparison

SOCP Relaxation

x* = argmin 12

∑ ui (1 + xi) + 14

∑ xi = 2 - |L|i Va

xi [-1,1]

Xii = 1 X - xxT 0

Derive SOCP relaxation from the SDP relaxation

Further Relaxation

2-D Example

X11 X12

X21 X22

x1x1 x1x2

x2x1 x2x2

xxT =x1

2 x1x2

2-D Example(X - xxT)

1 - x12 X12-x1x2

X12-x1x2 1 - x22

C1 0 C1 0

(x1 + x2)2 2 + 2X12

SOC of the form || v ||2 st

2-D Example(X - xxT)

1 - x12 X12-x1x2

X12-x1x2 1 - x22

C2 0 C2 0

(x1 - x2)2 2 - 2X12

SOCP Relaxation

Consider a matrix C1 = UUT 0

(X - xxT)

||UTx ||2 X . C1

C1 . 0

Continue for C2, C3, … , Cn

Kim and Kojima, 2000

SOCP Relaxation

How many constraints for SOCP = SDP ?

Exponential.

Specify constraints similar to the 2-D example

(xi + xj)2 2 + 2Xij

(xi + xj)2 2 - 2Xij

SOCP-MS

x* = argmin 12

∑ ui (1 + xi) + 14

∑ xi = 2 - |L|i Va

xi [-1,1]

Xii = 1 X - xxT 0

Muramatsu and Suzuki, 2003

SOCP-MS

x* = argmin 12

∑ ui (1 + xi) + 14

∑ xi = 2 - |L|i Va

xi [-1,1]

(xi + xj)2 2 + 2Xij (xi - xj)2 2 - 2Xij

Specified only when Pij 0

Muramatsu and Suzuki, 2003

Outline

• Comparison

Dominating Relaxation

For all MAP Estimation problem (u, P)

A dominates B

Dominating relaxations are better

Equivalent Relaxations

A dominates B

B dominates A

Strictly Dominating Relaxation

A dominates B

B does not dominate A

SOCP-MS

(xi + xj)2 2 + 2Xij (xi - xj)2 2 - 2Xij

min ij Pij Xij

• Pij ≥ 0(xi + xj)2

2- 1Xij =

• Pij < 0(xi - xj)2

21 -Xij =

SOCP-MS is a QP

Same as QP by Ravikumar and Lafferty, 2006

SOCP-MS ≡ QP-RL

LP-S vs. SOCP-MSDiffer in the way they relax X = xxT

Xij [-1,1]

1 + xi + xj + Xij ≥ 0

∑ Xij = (2 - |L|) xij Vb

(xi + xj)2 2 + 2Xij

(xi - xj)2 2 - 2Xij

SOCP-MS

F(LP-S)

F(SOCP-MS)

LP-S vs. SOCP-MS

• LP-S strictly dominates SOCP-MS

• LP-S strictly dominates QP-RL

• Where have we gone wrong?

• A Quick Recap !

Recap of SOCP-MS

Can we use different C matrices ??

Can we use a different subgraph ?? 1 -1

(xi - xj)2 2 - 2Xij

(xi + xj)2 2 + 2Xij

Outline

• Comparison

• Generalization of Results– SOCP Relaxations on Trees– SOCP Relaxations on Cycles

SOCP Relaxations on Trees

Choose any arbitrary tree

SOCP Relaxations on Trees

Choose any arbitrary C 0

Repeat over trees to get relaxation SOCP-T

LP-S strictly dominates SOCP-T

LP-S strictly dominates QP-T

Outline

• Comparison

• Generalization of Results– SOCP Relaxations on Trees– SOCP Relaxations on Cycles

SOCP Relaxations on Cycles

Choose an arbitrary even cycle

Pij ≥ 0 Pij ≤ 0OR

Choose any arbitrary C 0

Repeat over even cycles to get relaxation SOCP-E

LP-S strictly dominates SOCP-E

LP-S strictly dominates QP-E

• True for odd cycles with Pij ≤ 0

• True for odd cycles with Pij ≤ 0 for only one edge

• True for odd cycles with Pij ≥ 0 for only one edge

• True for all combinations of above cases

Outline

• Comparison

• Two New SOCP Relaxations– The SOCP-C Relaxation– The SOCP-Q Relaxation

The SOCP-C Relaxation Include all LP-S constraints True SOCP

Cycle of size 4

Define SOCP Constraint using appropriate C

SOCP-C strictly dominates LP-S

SOCP-C strictly dominated by cycle inequalities?

Open Question !!!

Outline

• Comparison

• Two New SOCP Relaxations– The SOCP-C Relaxation– The SOCP-Q Relaxation

The SOCP-Q Relaxation Include all cycle inequalities True SOCP

Clique of size n

Define an SOCP Constraint using C = 1

SOCP-Q strictly dominates LP-S

SOCP-Q strictly dominates cycle inequalities

Conclusions

• Large class of SOCP/QP dominated by LP-S

• New SOCP relaxations dominate LP-S

• Preliminary experimental results in poster

Future Work

• Comparison with cycle inequalities

• Determine best SOC constraints

• Develop efficient algorithms for new relaxations

Questions ??

4-Neighbourhood MRF

Test SOCP-C

50 binary MRFs of size 30x30

u ≈ N (0,1)

P ≈ N (0,σ2)

4-Neighbourhood MRF

σ = 2.5

8-Neighbourhood MRF

Test SOCP-Q

50 binary MRFs of size 30x30

u ≈ N (0,1)

P ≈ N (0,σ2)

8-Neighbourhood MRF

σ = 1.125

Equivalent Relaxations

A dominates B

B dominates A

For all MAP Estimation problem (u, P)

Strictly Dominating Relaxation

A dominates B

B does not dominate A

For at least one MAP Estimation problem (u, P)

An Analysis of Convex Relaxations M. Pawan Kumar Vladimir Kolmogorov Philip Torr for MAP Estimation.

Documents