Date post: | 28-Jun-2015 |
Category: |
Technology |
Upload: | yandex |
View: | 983 times |
Download: | 5 times |
C. Olsson
Higher-order Segmentation Functionals: Entropy, Color Consistency, Curvature, etc.
Yuri Boykov jointly with
O. Veksler
Andrew Delong
L. Gorelick
C. Nieuwenhuis E. Toppe
I. Ben Ayed M. Tang
A. Delong
H. Isack
A. Osokin
Different surface representations
mesh level-sets graph labeling
on complex
on grid
point cloud labeling
ps
continuous optimization
mixed optimization
Zps p{0,1}ps
combinatorial optimization
this talk
graph labeling
{0,1}ps
combinatorial optimization
Implicit surfaces/bondary
on grid
Sf,E(S) B(S)Sf,E(S)
Image segmentation Basics
4 I
Fg)|Pr(I Bg)|Pr(I
p
pp sfE(S)
bg)|Pr(I
fg)|Pr(Ilnf
p
p
p
S
{0,1}ps
Linear (modular) appearance of region
p
pp sfSfSR ,)(
Examples of potential functions
• Log-likelihoods • Chan-Vese • Ballooning
2)( cIf pp
1pf
)( pp If Prln
Basic boundary regularization for
{0,1}ps
pair-wise discontinuities
][)( qp
Npq
sswSB
Basic boundary regularization for
{0,1}ps
second-order terms
qpqpqp ssssss )()(][ 11
quadratic
][)( qp
Npq
sswSB
Basic boundary regularization for
{0,1}ps
Examples of discontinuity penalties
• Boundary length • Image-weighted boundary length
1pqw
2)( qppq IIw exp
second-order terms
][)( qp
Npq
pq sswSB
Basic boundary regularization for
• corresponds to boundary length | |
– grids [B&K, 2003], via integral geometry
– complexes [Sullivan 1994]
• submodular second-order energy
– can be minimized exactly via graph cuts [Greig et al.’91, Sullivan’94, Boykov-Jolly’01]
pqw
n-links
s
t a cut
{0,1}ps
second-order terms
][)( qp
Npq
pq sswSB
2
any (binary) segmentation energy E(S) is a set function E: S
Ω
Submodular set functions
Submodular set functions
Set function is submodular if for any 2:E
)()()()( TESETSETSE TS,
Significance: any submodular set function can be globally optimized in polynomial time
[Grotschel et al.1981,88, Schrijver 2000]
S T Ω
)||( 9O
Submodular set functions
Set function is submodular if for any 2:E
)()}{()()}{( SEvSETEvTE
TS
S T Ω
an alternative equivalent definition providing intuitive interpretation: “diminishing returns”
v v
Easily follows from the previous definition: E(T))}{()()}{( vSESEvTE
STS TS
Significance: any submodular set function can be globally optimized in polynomial time
[Grotschel et al.1981,88, Schrijver 2000] )||( 9O
Graph cuts for minimization of
submodular set functions
Assume set Ω and 2nd-order (quadratic) function
Function E(S) is submodular if for any
)()()()( 10011100 ,,,, pqpqpqpq EEEE
Nqp )( ,
Significance: submodular 2nd-order boolean (set) function can be globally optimized in polynomial time by graph cuts
[Hammer 1968, Pickard&Ratliff 1973]
Npq
qppq ssEsE)(
, )()( }{ 10,, qp ssIndicator variables
[Boros&Hammer 2000, Kolmogorov&Zabih2003] )|||N(| 2O
Combinatorial optimization
Continuous optimization
submodularity convexity
Global Optimization
?
Assume Gibbs distribution over binary random variables
for
Graph cuts for minimization of posterior energy (MRF)
}{ 10,ps
))((),...,( SEexpssPr n1
Theorem [Boykov, Delong, Kolmogorov, Veksler in unpublished book 2014?]
All random variables sp are positively correlated iff set function E(S) is submodular
That is, submodularity implies MRF with “smoothness” prior
}{ 1s|pS p
Basic segmentation energy
][ qp
Npq
pqp
p
p sswsf
boundary smoothness segment region/appearance
this talk
Higher-order binary segmentation
Curvature (3-rd order)
Convexity (3-rd order)
segment region/appearance
Shape priors (N-th order) Connectivity (N-th order)
Cardinality potentials (N-th order)
Appearance Entropy (N-th order)
Color consistency (N-th order)
Distribution consistency (N-th order)
boundary smoothness
submodular approximations [our work: Trust Region 13, Auxiliary Cuts 13]
global minimum [our work: One Cut 2014]
block-coordinate descent [Zhu&Yuille 96, GrabCut 04]
Overview of this talk
• From likelihoods to entropy
• From entropy to color consistency
• Convex cardinality potentials
• Distribution consistency
• From length to curvature
optimization high-order functionals
other extensions [arXiv13]
][)Pr(Npq
qppq
p
sp sswISEp
|ln),|( 10
assuming known
[Boykov&Jolly, ICCV2001]
image segmentation, graph cut
RGBI p
• parametric models – e.g. Gaussian or GMM • non-parametric models - histograms
}{ 10,ps
pair-wise (quadratic) term unary (linear) term
Given likelihood models
guaranteed globally optimal S
Beyond fixed likelihood models
[Rother, et al. SIGGRAPH’2004]
iterative image segmentation, Grabcut (block coordinate descent )
RGBI p
10 ,S
Models 0 , 1 are iteratively re-estimated
(from initial box)
extra variables • parametric models – e.g. Gaussian or GMM • non-parametric models - histograms
}{ 10,ps][)Pr(Npq
qppq
p
sp sswISEp
|ln),,( 10
pair-wise (quadratic) term mixed optimization term
NP hard mixed optimization! [Vesente et al., ICCV’09]
• Minimize over segmentation S for fixed 0 , 1
• Minimize over 0 , 1 for fixed labeling S
Block-coordinate descent for
fixed for S=const
)( 10 ,,SE
Npq
qppq
p
Sp sswISEp
][)Pr()( |ln,, 10
Npq
qppq
sp
p
sp
p sswIISEpp
][)Pr()Pr()(1
1
0
010
::
|ln|ln,,
Sp0ˆ Sp1
ˆ
distribution of intensities in current bkg. segment ={p:Sp=0}
distribution of intensities in current obj. segment S={p:Sp=1} S
optimal S is computed using graph cuts, as in [BJ 2001]
Iterative learning of color models (binary case )
• GrabCut: iterated graph cuts [Rother et al., SIGGRAPH 04]
start from models 0 , 1 inside and outside some given box
iterate graph cuts and model re-estimation until convergence to a local minimum
][)Pr()(Npq
qppq
p
Sp sswISEp
|ln,, 10
}{ 10,ps
solution is sensitive to initial box
BCD minimization of converges to a local minimum )( 10 ,,SE
E=2.37×106 E=2.41×106 E=1.39×106 E=1.410×106
Iterative learning of color models (binary case ) }{ 10,ps
(interactivity a la “snakes”)
Iterative learning of color models (could be used for more than 2 labels )
• Unsupervised segmentation [Zhu&Yuille, 1996]
][)Pr()(Npq
qppq
p
Sp sswISEp
|ln...,,, 210 || labels
}{ ,...,, 210ps
using level sets + merging heuristic
initialize models 0 , 1 , 2 , from many randomly sampled boxes
iterate segmentation and model re-estimation
until convergence
models compete, stable result if sufficiently many
Iterative learning of color models (could be used for more than 2 labels )
• Unsupervised segmentation [Delong et al., 2012]
|| labels
}{ ,...,, 210ps
using a-expansion (graph-cuts)
initialize models 0 , 1 , 2 , from many randomly sampled boxes
models compete, stable result if sufficiently many
iterate segmentation and model re-estimation
until convergence
][)Pr()(Npq
qppq
p
Sp sswISEp
|ln...,,, 210
Iterative learning of other models (could be used for more than 2 labels )
• Geometric multi-model fitting [Isack et al., 2012]
initialize plane models 0 , 1 , 2 , from many randomly sampled SIFT matches
in 2 images of the same scene
|| labels
using a-expansion (graph-cuts)
iterate segmentation and model re-estimation
until convergence
models compete, stable result if sufficiently many
][-)(Npq
qppq
p
S sswppSEp
...,,, 210
}{ ,...,, 210ps
Iterative learning of other models (could be used for more than 2 labels )
• Geometric multi-model fitting [Isack et al., 2012]
initialize Fundamental matrices 0 , 1 , 2 , from many randomly sampled SIFT matches
in 2 consecutive frames in video
|| labels
using a-expansion (graph-cuts)
iterate segmentation and model re-estimation
until convergence
models compete, stable result if sufficiently many
VIDEO
}{ ,...,, 210ps
][-)(Npq
qppq
p
S sswppSEp
...,,, 210
From color model estimation to entropy and color consistency
global optimization in One Cut
[Tang et al. ICCV 2013]
Interpretation of log-likelihoods: entropy of segment intensities
S
S1
Si
S2
S3
S4 S5
pixels of color i in S
}|{ iISpS pi
||
||
S
Sp is
i
probability of intensity i in S
Sp
pI )( |Prln
i
i
S
i ppS ln||
=
where = {p1 , p2 , ... , pn }
given distribution of intensities
},...,,{ S
n
SSS pppp 21
distribution of intensities observed at S
cross entropy
of distribution pS w.r.t.
H(S| )
i
ii pS ln||
=
Interpretation of log-likelihoods: entropy of segment intensities
][)()()(Npq
qppq sswSH|S|SH|S|SE
Npq
qppq
1S:p
1p
0S:p
0p ssw|Iln|Ilnpp
][)Pr()Pr()( 10 ,,SE
)( 0||| SHS )( 1||| SHS
10 ,min
entropy of intensities in S
entropy of intensities in S
minimization of segments entropy
Note: H(P|Q) H(P) for any two distributions (equality when Q=P) cross-entropy entropy
joint estimation of S and color models [Rother et al., SIGGRAPH’04, ICCV’09]
[Tang et al, ICCV 2013]
Interpretation of log-likelihoods: entropy of segment intensities
)( 10 ,,SE
)( 0||| SHS )( 1||| SHS
10 ,min
entropy of intensities in S
entropy of intensities in S
binary optimization
Note: H(P|Q) H(P) for any two distributions (equality when Q=P) cross-entropy entropy
mixed optimization
[Tang et al, ICCV 2013]
[Rother et al., SIGGRAPH’04, ICCV’09]
][)()()(Npq
qppq sswSH|S|SH|S|SE
Npq
qppq
1S:p
1p
0S:p
0p ssw|Iln|Ilnpp
][)Pr()Pr(
][)()()(Npq
qppq sswSH|S|SH|S|SE
Interpretation of log-likelihoods: entropy of segment intensities
)( 10 ,,SE
)( 0||| SHS )( 1||| SHS
10 ,min
entropy of intensities in S
entropy of intensities in S
common energy for categorical clustering, e.g. [Li et al. ICML’04]
Note: H(P|Q) H(P) for any two distributions (equality when Q=P) cross-entropy entropy
Npq
qppq
1S:p
1p
0S:p
0p ssw|Iln|Ilnpp
][)Pr()Pr(
Minimizing entropy of segments intensities (intuitive motivation)
][)()()(Npq
qppq sswSH|S|SH|S|SE
unsupervised image segmentation (like in Chan-Vese)
high entropy segmentation
break image into two coherent segments with low entropy of intensities
S
S
low entropy segmentation
S
S S
S S
S
more general than Chan-Vese (colors can vary within each segment)
S
S
S
S
break image into two coherent segments with low entropy of intensities
Minimizing entropy of segments intensities (intuitive motivation)
][)()()(Npq
qppq sswSH|S|SH|S|SE
From entropy to color consistency
all pixels i
Minimization of entropy encourages pixels i of the same color bin i to be segmented together
(proof: see next page)
i
12
4
3
5
From entropy to color consistency
][)()()(Npq
qppq sswSH|S|SH|S|SE
S
i
i
S
i
S
i
i
S
i ppSppS ln||ln||
||
||
||
||ln||ln||
S
S
i
iS
S
i
iii SS
||ln||||ln|| i
i
i
i
i SSSS ||ln||||ln|| i
i
i
i
i SSSS
||ln||||ln|| SSSSi
iiii |S|ln|S||S|ln|S| )(
volume balancing
color consistency
|S| |S|
S
S
Si Si
Si = S i
|S| |Si|
i i
pixels in each color bin i prefer to be together (either inside object
or background)
From entropy to color consistency
||ln||||ln|| SSSS
volume balancing
color consistency
S
Si = S i
|S| |Si|
i i
segmentation S with better
color consistency
pixels in each color bin i prefer to be together (either inside object
or background)
S
i
iiii |S|ln|S||S|ln|S| )(
From entropy to color consistency
||ln||||ln|| SSSS
volume balancing
color consistency
S
Si = S i
|S| |Si|
i i
convex function of cardinality |S|
(non-submodular)
pixels in each color bin i prefer to be together (either inside object
or background)
S concave function of
cardinality |Si| (submodular)
Graph-cut constructions for similar cardinality terms (for superpixel consistency)
[Kohli et al. IJCV’09]
In many applications, this term can be either dropped or replaced
with simple unary ballooning [Tang et al. ICCV 2013]
i
iiii |S|ln|S||S|ln|S| )(
|Si|
From entropy to color consistency
||ln||||ln|| SSSS
volume balancing
color consistency
(also, simpler construction)
connect pixels in each color bin to corresponding auxiliary nodes
][Npq
qppq ssw
boundary smoothness
|S| |Si|
i i
In many applications, this term can be either dropped or replaced
with simple unary ballooning [Tang et al. ICCV 2013]
convex function of cardinality |S|
(non-submodular)
L1 color separation works better in practice [Tang et al. ICCV 2013]
i
iiii |S|ln|S||S|ln|S| )(
smoothness + color consistency
One Cut [Tang, et al., ICCV’13]
connect pixels in each color bin to corresponding auxiliary nodes
Grabcut is sensitive to bin size
guaranteed global minimum
bo
x se
gmen
tati
on
linear ballooning
inside the box
smoothness + color consistency
One Cut [Tang, et al., ICCV’13]
bo
x se
gmen
tati
on
ballooning from hard constraints
linear ballooning from
saliency measure
connect pixels in each color bin to corresponding auxiliary nodes
guaranteed global minimum linear ballooning
inside the box
fro
m s
eed
s sa
lien
cy-b
ased
se
gmen
tati
on
photo-consistency + smoothness + color consistency
Color consistency can be integrated into
binary stereo
connect pixels in each color bin to corresponding auxiliary nodes
+ color consistency
photo-consistency+smoothness
Approximating: - Convex cardinality potentials - Distribution consistency - Other high-order region terms
d||SS| | 0
min
General Trust Region Approach (overview)
Trust region
(S)(S)E(S) BH
(S)(S)(S)E BU0
~
1st-order approximation for H(S)
0S
dsubmodular
(easy) hard
||SS||λB(S)(S)U(S)L 00λ
• Constrained optimization
minimize
• Unconstrained Lagrangian Formulation
minimize
d||SS||s.t.
B(S)(S)U(S)E
0
0
~
can be approximated with unary terms [Boykov,Kolmogorov,Cremers,Delong, ECCV’06]
45
General Trust Region Approach (overview)
Approximating L2 distance
0C
2
s dsdCdC,dC
|||| 0SS
p
o
ppp ssd2 )(
unary potentials [Boykov et al. ECCV 2006]
C
p dpd2
dp - signed distance map from C0
Trust Region Approximation
|S|
][)Pr(Npq
qppq
p
Sp ssw|Ilnp
submodular terms
appearance log-likelihoods boundary length
non-submodular term
volume constraint
Linear approx. at S0
S0
S0 submodular approx.
trust region
p
o
ppp ssd )(
L2 distance to S0
Volume Constraint for Vertebrae segmentation
Log-Lik. + length
48
Back to entropy-based segmentation
Interactive segmentation
with box
volume balancing
color consistency
][Npq
qppq ssw
boundary smoothness
|S| |Si|
i i
+ +
submodular terms non-submodular term
global minimum
Approximations (local minima near the box)
Trust Region Approximation
Surprisingly, TR outperforms QPBO, DD, TRWS, BP, etc.
on many high-order [CVPR’13] and/or
non-submodular problems [arXiv13]
Curvature
Pair-wise smoothness: limitations
52
• discrete metrication errors
4-neighborhood
- continuous convex formulations
8-neighborhood
- resolved by higher connectivity
Pair-wise smoothness: limitations
53
• boundary over-smoothing (a.k.a. shrinking bias)
Pair-wise smoothness: limitations
54
- curvature
- needs higher-order smoothness
• boundary over-smoothing (a.k.a. shrinking bias)
multi-view reconstruction
[Vogiatzis et al. 2005]
Higher-order smoothness & curvature for discrete regularization
• Geman and Geman 1983 (line process, simulated annealing)
• Second-order stereo and surface reconstruction – Li & Zuker 2010 (loopy belief propagation) – Woodford et al. 2009 (fusion of proposals, QPBO) – Olsson et al. 2012-13 (fusion of planes, nearly submodular)
• Curvature in segmentation: – Schoenemann et al. 2009 (complex, LP relaxation, many extra variables) – Strandmark & Kahl 2011 (complex, LP relaxation,…) – El-Zehiry & Grady 2010 (grid, 3-clique, only 90 degree accurate, QPBO) – Shekhovtsov et al. 2012 (grid patches, approximately learned, QPBO) – Olsson et al. 2013 (grid patches, integral geometry, partial enumeration) – Nieuwenhuis et al 2014? (grid, 3-cliques, integral geometry, trust region)
this talk good approximation of curvature, better and faster optimization practical !
the rest of the talk:
• Absolute curvature regularization on a grid [Olsson, Ulen, Boykov, Kolmogorov - ICCV 2013]
• Squared curvature regularization on a grid [Nieuwenhuis, Toppe, Gorelick, Veksler, Boykov - arXiv 2013]
Absolute Curvature
dsS
||
Motivating example: for any convex shape 2dsS
||
• no shrinking bias • thin structures
Absolute Curvature
n
n
easy to estimate via approximating
polygons
dsS
||
polygons also work for p [Bruckstein et al. 2001]
curvature on a cell complex (standard geometry)
/2 /4
/4
/2
• Schoenemann et al. 2009 • Strandmark & Kahl 2011
4- or 3-cliques on a cell complex
solved via LP relaxations
curvature on a cell complex (standard geometry)
/2 /4
/4
/2
cell-patch cliques on a complex
• Olsson et al., ICCV 2013
partial enumeration + TRWS
zero gap
reduction to pair-wise Constrain Satisfaction Problem
- new graph: patches are nodes - curvature is a unary potential
- patches overlap, need consistency
- tighter LP relaxation P4 P5 P6
P1 P2 P3
A B C D E F G H
curvature on a cell complex (standard geometry)
0 0 0 0 0 0 0 0 0 0 0
/2 /4
/4
/2
A
A
A
A
B
C
F
F
G
H
E
D 0
0
0
0
2A+B= /2 A+F+G+H = /4
D+E+F = /2
A+C= /4
/4 /2 /2 /2 /4
curvature on a pixel grid (integral geometry)
representative cell-patches representative pixel-patches
2x2 patches 3x3 patches 5x5 patches
zero gap
integral approach to absolute curvature
on a grid
integral approach to absolute curvature
on a grid
2x2 patches 3x3 patches 5x5 patches
zero gap
Squared Curvature with 3-cliques
dsS
2
S
S 3-cliques
with configurations (0,1,0) and (1,0,1)
p p+
p-
general intuition example
Nieuwenhuis et al., arXiv 2013
more responses where curvature is higher
N
n
nn
C
sdss1
22 )(
Δ i
Δ i
1
2
3
…
n
N
N-1
n-1
n+1
n+2
…
…
C
rn
Δ i
5x5 neighborhood
ci i
N
n
nin
1
)(||
)(|| ninn s
rn
n
4
3ddR ||, )(
d
r =1/
zoom-in
Thus, appropriately weighted 3-cliques estimate squared curvature integral
r
rds
rCircle
12
)(
Experimental evaluation
Experimental evaluation
r
rds
rCircle
12
)(
Model is OK on given segments. But, how do we optimize non-submodular
3-cliques (010) and (101)?
1. Standard trick: convert to non-submodular pair-wise binary optimization
2. Our observation: QPBO does not work (unless non-submodular regularization is very weak)
Fast Trust Region [CVPR13, arXiv]
uses local submodular approximations
Segmentation Examples
length-based regularization
elastica [Heber,Ranftl,Pock, 2012]
Segmentation Examples
90-degree curvature [El-Zehiry&Grady, 2010]
Segmentation Examples
our squared curvature
7x7 neighborhood Segmentation Examples
our squared curvature (stronger)
7x7 neighborhood Segmentation Examples
our squared curvature (stronger)
2x2 neighborhood Segmentation Examples
Binary inpainting length squared curvature
Conclusions • Optimization of Entropy is a useful information-
theoretic interpretation of color model estimation
• L1 color separation is an easy-to-optimize objective useful in its own right [ICCV 2013]
• Global optimization matters: one cut [ICCV13]
• Trust region, auxiliary cuts, partial enumeration
General approximation techniques
- for high-order energies [CVPR13]
- for non-submodular energies [arXiv’13]
outperforming state-of-the-art combinatorial optimization methods