Graph Cuts - The College of Engineering at the University of Utahcs6320/cv_files/GraphCuts.pdf ·...

Graph CutsSrikumar Ramalingam

School of Computing

University of Utah

Outline

• Introduction

• Pseudo-Boolean Functions

• Submodularity

• Max-flow / Min-cut Algorithm

• Alpha-Expansion

Segmentation Problem

[Boykov and Jolly’2001,Rother et al. 2004]

Stereo Reconstruction

• Choose the disparities from the discrete set: ),...,2,1( L

Image Denoising

Original Denoised

image

Semantic Labeling (Building, ground, sky)

[Hoiem, Efros, Hebert,

IJCV, 2007 ]

Image Labeling Problems

Labeling is highly structured

Highly unlikely

Image Courtesy: Lubor Ladicky

Labeling is highly structured

Slide Courtesy: Lubor Ladicky

Image Labeling Problems

Slide Courtesy: Lubor Ladicky

Outline

• Introduction


• Submodularity


• Alpha-Expansion

Pseudo Boolean Functions (PBF)• Variables: 1,0,...,, 21 nxxx

[Boros&Hammer’2002]

• Negations: 1,01 ii xx

• Pseudo-Boolean Functions (PBF):

» Maps a Boolean vector to a real number.

Rf n }1,0{:

• Has unique multi-linear representation:

» For example:

321424321 532),,,( xxxxxxxxxf

Posiforms for Pseudo-Boolean functions (PBF)• Posiforms: Non-negative multi-linear polynomial except

maybe the constant terms.

13

32142 5)1(32 xxxxx

321424 5331 xxxxxx


• Several posiforms exist for a given function.

321424 53)1(32 xxxxxx

321424321 532),,,( xxxxxxxxxf

321424 5332 xxxxxx

• Provides bounds for minimization, e.g. 1

Set Functions are Pseudo Boolean Functions (PBF)

• Finite ground set },...,2,1{ nV

• Set function (Input - subset of , output - real number)V

Rf V

s 2:

• 1-1 correspondence exists between

and subset of

1,0,...,, 21 nxxx

.VS

1ix Si

0ix Si)4,2,1(}1,0,1,1{

}4,3,2,1{

4321

xxxx

V

Set Functions are Pseudo Boolean Functions (PBF)

• Consider a PBF32424321 532),,,( xxxxxxxxf

• Equivalent to a set function

2)0)(1(5)0)(1(32})2,1({ sf

7)1)(1(5)0)(1(32})3,2({ sf

Outline

• Introduction


• Submodularity


• Alpha-Expansion

Submodular set functions (Union-Intersection)

• A set function is submodular if and only if:

VBABAfBAfBfAf ,),()()()(

Rf V 2:

VBA ,

Equivalent Definitions

Slide Courtesy: Krause, Jegelka

Questions


Submodularity Example








Set cover is submodular


Submodular set functions (Union-Intersection)

VBABAfBAfBfAf ,),()()()(

Let us consider a very simple case with only two variables and1x .2x

}2{},1{},2,1{ BAV

Using submodularity, we have:

)0,0()1,1()1,0()0,1(

)0,0()1,1(

)1,0()0,1(

2121

2121

ffff

xxfxxf

xxfxxf

)0,0(f )1,0(f

)0,1(f )1,1(f

• Main diagonal elements are smaller than off-diagonal ones.

• Blue is larger than red.

Quadratic Pseudo Boolean Functions (QPBF)

• Example of quadratic pseudo Boolean functions

4321214321 531),,,( xxxxxxxxxxf


Submodular Quadratic Pseudo Boolean Functions

• A QPBF is submodular if and only if all quadratic coefficients are non-positive.

3221213213 5315),,( xxxxxxxxxf

Example for submodular QPBF

}3,2{},2,1{},3,2,1{ BAV

}2{},3,2,1{ BABA

7)1)(1(5)1)(0(3)1(3015)( Bf

13)0)(1(5)0)(1(3)1(3115)( Af

5)1)(1(5)1)(1(3)1(3115)( BAf

12)0)(1(5)0)(0(3)1(3015)( BAf

3231213213 53315),,( xxxxxxxxxf

)125713(),()()()( BAfBAfBfAf

Outline

• Introduction


• Submodularity


• Alpha-Expansion

Max-flow/Min-cut

Image courtesy: Lubor Ladicky

Max-flow/Min-cut

Max-flow/Min-cut

Max-flow/Min-cut

s

t

• A submodular QPBF can be associated with a network .f vG

tx2

1xs

1x

2x

• There is 1-1 correspondence every edge in network and

every term in .f

• Let us denote source by and sink by 0s .1t

• An edge that goes from to is denoted by 1x 2x .21xx

21xx1x 2x

Network model for submodular QPBF


2121 43 xxxxf

• Given a QPBF we rewrite it using a posiform representation

using only three types of terms:

s ttxx 22 11 xsx

1x 2x21xx

1x 2x

,jixx ,ix ,ix

)444(3 222121 xxxxxxf

2121 4333 xxtxxsf

22121 4)1(43 xxxxxf

2121 433 xxxxf

2121 4)333(3 xxxxf

2121 4)1(333 xxxxf

s

t

1x 2x

3

4

3

• There is a one-one correspondence between values of and s-t cut values of [Hammer 1965]


f .vG

4}),({

)1,0(

21

21

xxC

xxf

s

t

3

3

1x 2x4

s-t mincut[Ford&Fulkerson’62,Goldberg&Tarzan86]

21xx1xs

tx2

• There is a one-one correspondence between values of and s-t cut values of [Hammer 1965]


f .vG

s

t

3

3

1x 2x4

633}),{},,({

)0,1(

12

21

txsxC

xxf

f.vG

s-t mincut[Ford&Fulkerson’62,Goldberg&Tarzan86]

Thus we can compute the minimum of using maxflow/mincut algorithm on the associated

Network model for non-submodular QPBF

• There is no polynomial-time algorithm for s-t mincut on a network with negative edge capacities.

• A submodular QBPF can always be associated with a network with non-negative edge capacities.

• A non-submodular QBPF can be associated with a

network as follows:

f

vG

2121

2121

2121

453

)1(453

43

xxxsxsf

xxxxf

xxxxf

s

t

3 5

1x 2x-4

Minimizing Quadratic Pseudo Boolean Functions

• If QPBF is submodular, use maxflow algo.. [Ford&Fulkerson’62,Goldberg&Tarzan86]


• If QPBF is non-submodular, Belief propagation or other

message passing algorithms.

Multi-label Problems

• Choose the disparities from the discrete set: ),...,2,1( L

Multi-label ProblemsExact Methods:

Transform the given multi-label problems to Boolean problems and solve them using maxflow/mincut algorithms or QPBO techniques.

[Not covered in this course!]

Approximate Methods:

Develop iterative move-making algorithms where each move corresponds to a Boolean problem.

Outline

• Introduction


• Submodularity


• Alpha-Expansion

Boolean Energy Function• Variables .1,0,...,, 21 nxxx

j

xi }.1,0{ jxi

lm

xx ji lxi .mx j

1

0

1

0

1

0

1

0

21 21212211),(

i j

j

x

i

x

ij

xx

j

j

x

j

x

j

j

x

j

xxxE

- cost of assigning

- cost of jointly assigning and

Energy function:

Multi-label Energy Function• Variables

mm/dd/yy 44

.,...,1,0,...,, 21 Lyyy m

l

yi

lyi

l

yi .lyi

lm

yy ji lyi .my j

L

i

L

j

j

y

i

y

ij

yy

L

j

j

y

j

y

L

j

j

y

j

yyyE1 111

21 21212211),(

1

0 otherwise.

- cost for assigning a single variable

- cost of jointly assigning and

Energy function:

Move Making Algorithms

Solution Space

En

erg

y

[Image courtesy: Pushmeet Kohli, Phil Torr]

Move Making Algorithms

Solution Space

En

erg

y

Search

Neighbourhood

Current Solution

Optimal Move

[Image courtesy: Pushmeet Kohli, Phil Torr]

Expansion

building

[Boykov et al. 2001]

Expansion• Let and be two adjacent variables whose labels are

not

iy jy

.

iy

jy

al

bl

retain

retain

.


In the move space, we compute if the two variables should retain the same labels or move to label

Expansion• In the move space, we use two Boolean variables and to

denote and respectively. The encoding is shown below:

iy jyix jx

1

0

ii

iai

xy

xly

1

0

jj

jaj

xy

xly

ba

ji

ll

yy a

ji

l

yy

ji yy

b

ji

l

yy

01

jixx

11

jixx

00

jixx

10

jixx

=


• Submodularity condition states that the sum of main diagonal elements is less than or equal to the sum of elements in the off-diagonal:

𝑙𝑖 𝑙𝑗

• Submodularity condition states that the sum of main diagonal elements is less than the sum of elements in the off-diagonal:

Expansion

[Boykov et al. 2001][Image courtesy: Lubor Ladicky]

Swap• The variables having the labels and can swap their labels

or retain their previous states.

mm/dd/yy 52

iy

jy

retain

retain


• In the move space, we use two Boolean variables and to denote and respectively. The encoding is shown below:

mm/dd/yy 53

Swap

iy jy

1

0

ii

ii

xy

xy

1

0

jj

jj

xy

xy

ix jx

ji yy

ji yy

ji yy

ji yy

01

jixx

11

jixx

00

jixx

10

jixx

=



1

0


mm/dd/yy 54

Swap

ji yy

ji yy

ji yy

ji yy

01

jixx

11

jixx

00

jixx

10

jixx

=

0

,0

,,

2121

21

abba

aa

ll

yy

ll

yy

ll

yy

ba Lll


• Semi-metric condition:

Foreground / Background Estimation using Graph Cuts

Rother et al. SIGGRAPH04[Graph cuts slides adapted from Lubor Ladicky]

Foreground / Background Estimation

Data term Smoothness term

Data term

Estimated using FG / BG colour models

Smoothness term

where

Intensity dependent smoothness

Neighborhood terms

Parameters that are manually set or learned from data

Delta function is 1 when the condition is satisfied, and 0 otherwise





How to solve this optimization problem?



How to solve this optimization problem?

• Transform into min-cut / max-flow problem

• Solve it using min-cut / max-flow algorithm

References

What energy functions can be minimized using graph cuts?http://www.cs.cornell.edu/~rdz/Papers/KZ-PAMI04.pdf

Boykov et al. Fast Approximate Energy Minimization via Graph Cuts, 1999http://www.cs.cornell.edu/rdz/Papers/BVZ-iccv99.pdf

Tutorial on Energy functions (Longer version of the slides shown in the class with more details)https://www.inf.ethz.ch/personal/ladickyl/DIYConstruct.pdf

Ishikawa’s tutorial on graph cutshttp://www.f.waseda.jp/hfs/PSIVT2009.pdf

http://www.cs.cornell.edu/rdz/Papers/BVZ-iccv99.pdf

http://www.f.waseda.jp/hfs/PSIVT2009.pdf

Thank You

Date post:	14-Apr-2020
Category:	Documents
Upload:	others
View:	3 times
Download:	0 times

Graph Cuts - The College of Engineering at the University of Utahcs6320/cv_files/GraphCuts.pdf ·...

Documents