Submodularization for Binary Pairwise Energies
Lena Gorelick
joint work with
O. Veksler
I. Ben Ayed
A. Delong
Y. Boykov
2
Νqp
xxp
pp qpxfE
,][1(x)
Example of Simple Binary Energy
Potts Modelf 1,0x
3
Binary Pairwise EnergyQuadratic Form
Submodular Energy global optimum with graphcut (Boros &
Hammer, 2002)
qp
qppqp
pp xxvxuE,
(x) 1,0x
pqpq vv 0
Potts Model
4
0, pqvpq
Non-Submodular Energy NP-hard
Binary Pairwise Energy Quadratic Form
const.)(,
qp
qppqp
pp xxvxuE x
Middlebury
Image credit: Carlos Hernandes
5
Standard Optimization Methods
General energy - NP-hardApproximate methods:
Global Linearization: QPBO, TRWS, SRMP (Kolmogorov et al. 2006, 2014)
Local Linearization: parallel ICM, IPFP (Leordeanu, 2009)
Message Passing: BP (Pearl 1989)
6
Related WorkGlobal Linearization
)(xE QPBO, TRWS, SRMP (Kolmogorov et al. 2006,
2014)
)(~min..
yyE
Cts
Linearize introducing large number ofvariables and constraintsSolve relaxed LPor its dual
Integrality Gap
*relaxedyRounding
*integer x
Related WorkIterative Local Linearization
7
parallel ICM (Leordeanu, 2009) large steps weak min
IPFP (Leordeanu, 2009) controls step size by relaxation Integrality Gap
)(xE
x
txEt(x)~
1tx
N}1,0{Bounded domain of discrete configurations
8
Local Submodular Approximation LSA
Local Submodular Approximation model
Non-linear
Two ways to control step size
Et(x)~
)(xE
x
tx1tx
N}1,0{Bounded domain of discrete configurations
9
Trust Region Local submodular approximation
Auxiliary Functions = Surrogate Functions = Upper Bounds = Majorize-Minimize Local submodular upper bound
Never leave the discrete domain
Iterative Optimization Framework
LSA-AUX
LSA-TR
10
Iterative Optimization FrameworkTrust Region:
Discrete High Order Energies Relaxed Quadratic Binary Energies Levenberg Marquardt
Auxiliary Functions=Surrogate Functions =Upper Bounds = Majorize-Minimize Discrete High Order Energies
Gorelick et al. 2012,2013
Ben Ayed et al. 2013
Olsson et al. 2008
Narasimhan & Bilmes 2005
Rother et al. 2006
Hartley & Zisserman 2004
11
Local Submodular ApproximationLSA
qp
qppqp
pp xxvxuE,
)(x
)()()( sup xxx EEE sub
+- x
tx)(xE
12
Local Submodular ApproximationLSA
)()()( sup xxx EEE sub
x
tx)(xE
13
Approximate around
Local Submodular ApproximationLSA
)()()( sup xxx EEE sub
)(xE tx
)(~ x(x) subt EE
Et(x)~
x
tx)(xE
14
Submodular functionLSA
Approximate around
Local Submodular ApproximationLSA
)()()( sup xxx EEE sub
)(xE tx
)()(~ xx(x) approxt
subt EEE
Linear Approximation
Et(x)~
x
tx)(xE
15
0,)(sup pqqppqpq
vxxvE x
Linear Approximation of the Supermodular Term
16
qppq xxv
Linear Approximation of the Supermodular Term
17
Linear Approximation of the Supermodular Term
0 xy
18
0
1
x
y
1
Linear Approximation of the Supermodular Term
1,0
0,0
1,1
0,1
0 xy
19
Linear Approximation of the Supermodular Term
0
1
1
x
y
1,0
0 xy
20
1
constyx 0
0
1
Linear Approximation of the Supermodular Term
x
y
1,0
0,0
1,1
Linear (Unary)
approximation
21
Linear Approximation of the Supermodular Term
0
1
1
constyvxu
x
y
22
LSA-TR:Trust Region Overview
)(xE
x
tx
)()()( sup xxx EEE sub
23
LSA-TR:Trust Region Overview
)(xE
x
tx
Et(x)~
Newton Step
)()(~ xx(x) approxt
subt EEE
1tx
LSA-TRTrust Region Sub-Problem
24
)(xE
x
tx
Et(x)~
Trust Region
)()(~ xx(x) approxt
subt EEE
Trust Region Sub-Problem
td ||||s.t. txx
24
NP-hard!Constrained Submodular Optimization
25
fixed in each iteration inversely related to trust region size adjusted based on quality of approximation
LSA-TR: Approximate TR sub-problem
||||)()(
tt
approxt
subt EEL
xxxx(x)
Unary TermsBoykov et al. 2006
Gorelick et al. 2013
t
Submodular
26
Experiments & Results
27
Experiments & Results:Deconvolution Binary De-convolution All pairwise terms supermodular
Original Img Convolved Convolved+Noise
?
Experiments & Results:Deconvolution
28
Noise:N(0,0.05)
SRMP
QPBOI
TRWS FTR-L
LBP
LSA-TR (0.3 sec.)E=21.13
LSA-AUX (0.04 sec)E=34.70
TRWS:5000 iter.E=65.07
LBP5000 iter.E=40.15
QPBO(0.1 sec.)
QPBO-I (0.2 sec.)E=66.44
IPFP(0.4 sec.)E=32.90
SRMP:5000 iter.E=39.06
Experiments & Results:Segmentation of Thin Structures
29
QPBO
QPBO-IE= -77.08
LBPE= -84.54
IPFPE= 163.25
Image
SRMP
Potts, v<0(submodula
r)
with edge repulsion, v>0(non-submodular)
TRWSE= -67.21
LSA-TRE= -175.05
LSA-AUXE= -120.03
SRMPE= -101.61
Repulsion = Reward different labels across high contrast edges
30
Experiments & Results:Inpainting dtf-chinesechar database
LSA-TRInput ImgGround Truth
Kappes et al., 2013
31
Experiments & Results:In-painting Chinese Characters
32
Curvature Regularization Efficient Squared Curvature model –
(Nieuwenhuis et al. 2014, poster on Friday)Potts Model Elastica
90-degree curvature
Heber et al. 2012
El-Zehiry&Grady, 2010
Our curvature Using LSA-TR
33
Summary Two novel discrete optimization methods
Simple, efficient, state-of-art results The code is available online -
http://vision.csd.uwo.ca/code/
Extensions: Find new applications▪ Convexity Shape Prior (in ECCV14)
Alternative optimization framework with LSA▪ Pseudo-Bounds (in ECCV14)
Please come by our poster