CVPR2009: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Miami Beach,...

CVPR2009: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Miami Beach, Florida. June 20-25.

Higher-Order Clique Reduction in Binary Graph Cut

Hiroshi Ishikawa

Nagoya City UniversityDepartment of Information and Biological Sciences


Contribution of this work

Reduce any higher-order binary MRF

into first order

Adds variables

Can also be used for multi-label energy, with the Fusion Move technique

CC

CCn XfXXEXE )(),,()( 1

),(

1 ),()() ,,(~

)(~

vuvuuv

Vvvvn XXhXgXXEXE mX,, mX,,


Energy Minimization

Close to Y Smooth

Given Y Find X

Assigns Xv (= 0 or 1) to each pixel v

All pixels Neighboringpixels

),(

),()()(vu

vuuvVv

vv XXhXgXE


Energy Minimization

Good (Low Energy) Bad (High Energy)

Better (Lower Energy) Worse (Higher Energy)

A B C D


Energy Minimization

Good (Low Energy) Bad (High Energy)

12 Bad 12 Bad

40 Good 40 Good


Energy Minimization

10 As

0 Ds

Better (Lower Energy) Worse (Higher Energy)

8 Bs

3 Cs

10 As

0 Ds

4 Bs

7 Cs

A B C D


Third Order (Clique up to 4 pixels)Third Order (Clique up to 4 pixels)

),,,(

),,,(tsvu

tsvuuvst XXXXk),,,(

),,,(tsvu

tsvuuvst XXXXk

Higher-Order Energy

),(

),()()(vu

vuuvVv

vv XXhXgXE

CC

CC XfXE )()(

First Order (Clique up to 2 pixels)

General Order

C : a set of cliques

CvvC XX )(

Clique

Clique

Clique

Clique


First-Order MRF Minimization

Graph cutsGreig et al. ’89Boykov et al. CVPR’98, PAMI2001(-exp.)Kolmogorov & Zabih. PAMI2004

Belief propagationFelzenszwalb & Huttenlocher. IJCV2006Meltzer et al. ICCV2005

Tree-reweighted message passingKolmogorov. PAMI2006


Higher-Order MRF Minimization

Graph cutsKolmogorov & Zabih. PAMI2004Freedman & Drineas. CVPR2005Woodford et al. CVPR2008Kohli et al. PAMI’08, Cremers&Grady ECCV’06Rother et al. CVPR2009Komodakis & Paragios. CVPR2009

Belief propagationLan et al. ECCV2006Potetz. CVPR2008


Higher-Order MRF Minimization

Graph cutsKolmogorov & Zabih. PAMI2004Freedman & Drineas. CVPR2005Woodford et al. CVPR2008Kohli et al. PAMI’08, Cremers&Grady ECCV’06Rother et al. CVPR2009Komodakis & Paragios. CVPR2009

Belief propagationLan et al. ECCV2006Potetz. CVPR2008


Functions of Binary Variables

Pseudo-Boolean function (PBF)Function of binary (0 or 1) variablesCan always write uniquely as a polynomial

One variable x : E0 (1x) + E1 x

Two variables x, y :

E00 (1x) (1y) + E01 (1x) y + E10 x (1y) + E11 x y

Three variables x, y, z :

E000 (1x) (1y) (1z) + E001 (1x) (1y) z +…+ E111 x y z

nth order binary MRF = (n+1)th degree PBF


2nd-Order (Cubic) Case

Kolmogorov & Zabih. PAMI2004Freedman & Drineas. CVPR2005Reduce cubic PBF into quadratic one using

) (max wxyzw

0 0 0x y z

0}2 ,0max{) (max0

ww

={0,1}

0 0 1 0}1 ,0max{) (max0

ww

0 1 1 0 max0

ww

1 1 1 1}1 ,0max{ max1

ww

2 zyx

2

1

0

1


2nd-Order (Cubic) Case

)2(max

zyxwxyzw

If a < 0

So, in a minimization problem, we can substitute

xyza )2( zyxwaby

)2(minmin

,,,,,

zyxwaxyza

wzyxzyx Thus

)2(min

zyxwaxyzaw

min


Higher-Order Case

)2(max

zyxwxyzw

)3(max

tzyxwxyztw

)4(max

utzyxwxyztuw

)1(max 11

dxxwxx dw

d

)1(minmin 11 dxxwaxxa dd

if a < 0


Higher-Order Case

For a > 0 and d > 3, nothing similar is known

→ our contribution

Imagine such a formula:


Higher-Order Case

For a > 0 and d > 3, nothing similar is known→ our contribution

Imagine such a formula:

Notice LHS is symmetric i.e., if we swap the value of two variables, LHS is unchanged

So RHS must be symmetric, too.

)degree 2()degree 1(min ndst

wxyztw min


Symmetric Polynomial

FactAny symmetric polynomial can be written as a polynomial in terms of elementarysymmetric polynomials.

If f (x, y, z, t) is quadratic symmetric, it can bewritten with a polynomial P(u,v) :

) , (),,,( Ptzyxf

ztytxtzxyzxy

tzyx ESPs

21 ss

2

1

s

s


Quartic (Degree 4) Case ) () (min

wxyzt

w degree 1st degree 2nd

min wxyztw

),( 21 ssQ)( 1sP

ztytxtzxyzxys

tzyxs

2

1

212

1 sss

)( 1sP

),( 21 ssQ

bsa 1


Quartic (Degree 4) Case ) () (min

wxyzt

w degree 1st degree 2nd

min wxyztw

),( 21 ssQ)( 1sP

2222222

1 2)( stzyxtzyxs

esdsc 21 ),( 21 ssQ

etc.) , , (since 2 2221 yyxxss

ztytxtzxyzxys

tzyxs

2

1

212

1 sss

)( 1sP

),( 21 ssQ

bsa 1


) (min wxyztw

) (min wxyztw

Quartic (Degree 4) Case

bsa 1 esdsc 21

ztytxtzxyzxy

tzyxww

3)(2min

32 1 s 2s

An exhaustive search for a, b, c, d, e yields

) () (min

wxyztw

degree 1st degree 2nd


Quintic (Degree 5) Case

Similarly,

and so on, until one can guess…

tuzuyuxuztytxtzxyzxyr

utzyxr

2

1

211),(

)3()32(min rrwrvxyztuwv


General Case

where

1

1 12

11 ,

d

i

d

ijji

d

ii xxSxS

21

1,,

1 1)2(min1

SiSkwxxd

dn

n

i

dii

wwd

otherwise2

and odd is 1 ddi

nidk

2

1dnd


General Case

For each monomial, the number of new variable is:

For instance, general quintic looks like:

So the number is exponential in degree

....2

432

xyzyzwtxzwt

xywtxyztxyzwxyzwt

2

1dnd


Multiple labels: Fusion Move

Assume labelsLabeling Y assigns a label Yv to each v

Fusion MoveIteratively update Y :

1. Generate a proposed labeling P

},,{ 1 NllL

Lempitsky et al. ICCV2007



Assume labelsLabeling Y assigns a label Yv to each v


1. Generate a proposed labeling P2. Merge Y and P

The merge defines a binary problem:

“For each v, change Yv to Pv or not”

Lempitsky et al. ICCV2007

},,{ 1 NllL







10

01

10

00

10

11

01

00

Y P X







10

01

10

00

10

11

01

00

Y P X

00

00

00

01

11

11

10

10


Fusion Move with QPBO

QPBO (Roof duality)

Minimizes submodular E globally.For non-submodular E, assigns each pixel

0, 1, or unlabeled

With fusion move, by not changing unlabeled pixels to P, E doesn’t increase

Hammer et al. 1984, Boros et al. 1991, 2006Kolmogorov & Rother PAMI2007, Rother et al. CVPR2007


Experiment: Denoising by FoE

FoE (Fields of Experts) Roth & Black CVPR2005

A higher-order prior for natural images

K

iCiiCC YJYf

1

2

2

11log)(

CC

CC YfYE )()( C : a set of cliques

CvvC YY )(

C :

C :2

2

}{}{ 2

)()(

vv

vv

YNYf



Original Noise-added 1st order3rd order



Lan et al. Potetz This work

= 10

= 20

0

5000

10000

15000

20000

25000

30000

35000

40000

45000

24

25

26

27

28

29

30

31

32

PSNR (larger the better)Energy (smaller the better)

Lan et al. Potetz This work

Lan et al. ECCV2006 ~8 hoursPotetz. CVPR2008 ~30 minsThis work ~10 mins



Energy & PSNRTwo proposal generation strategies

20

40

60

80

100

120

0 50 100 150 200 25022

23

24

25

26

27

0 50 100 150 200 250

PSNRE (×1000)

time (sec.)time (sec.)

blur & random

expansionblur & random

expansion


Summary

Reduce any higher-order binary MRF

into first order

Adds variablesNumber exponential in order

For multi-label, can be used with Fusion Move with QPBO

CC

CCn XfXXEXE )(),,()( 1

),(

1 ),()() ,,(~

)(~

vuvuuv

Vvvvn XXhXgXXEXE mX,, mX,,


Thank you!

Code available athttp://www.nsc.nagoya-cu.ac.jp/~hi/

AcknowledgementsStefan Roth, Brian Potetz, and

Vladimir Kolmogorov

Date post:	29-Mar-2015
Category:	Documents
Upload:	nora-spratley
View:	213 times
Download:	0 times

CVPR2009: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Miami Beach,...

Documents