Dr. Ulas Bagci bagci@ucfbagci/teaching/computervision16/Lec13.pdfDr. Ulas Bagci bagci@ucf.edu...

transcript

CAP5415-ComputerVisionLecture13-AdvancedImageSegmentation

(OptimizationProblem)

Dr.UlasBagcibagci@ucf.edu

Lecture13:AdvancedImageSegmentation

Reminders• Oct18GuestLecture(Lecture17)Dr.Borji

–BasicsofHumanVisionSystem• Oct20GuestLecture(Lecture18)Dr.Gong

–DeepLearning(CNN,RNN,LSTM,GRU)

• Selectyourprojectby15th ofOctober(UseWebCourse!)

Labeling &Segmentation• Labeling isacommonwayformodelingvariouscomputervisionproblems(e.g.opticalflow,imagesegmentation,stereomatching,etc)

• Thesetoflabelscanbediscrete(asinimagesegmentation)

L = {l1, . . . , lm} with L = m

• Thesetoflabelscanbediscrete(asinimagesegmentation)

• Orcontinuous(asinopticalflow)

L = {l1, . . . , lm} with L = m

L ⇢ Rnfor n � 1

Labelingisafunction• Labelsareassignedtosites (pixellocations)

Labelingisafunction• Labelsareassignedtosites (pixellocations)• Foragivenimage,wehave

|⌦| = Ncols

.Nrows

Labelingisafunction• Labelsareassignedtosites (pixellocations)• Foragivenimage,wehave• Identifyingalabelingfunction(withsegmentation)is

|⌦| = Ncols

.Nrows

f : ⌦ ! L

Weaimatcalculatingalabelingfunctionthatminimizesagiven(total)errororenergy

|⌦| = Ncols

.Nrows

f : ⌦ ! L

Weaimatcalculatingalabelingfunctionthatminimizesagiven(total)errororenergy

*Aisanadjacencyrelationbetweenpixellocations10

|⌦| = Ncols

.Nrows

f : ⌦ ! L

E(f) =X

[Edata

(p, fp

q2A(p)

Esmooth

EnergyFunctionTheroleofanenergyfunctioninminimization-basedvisionistwofold:

1. asthequantitativemeasureoftheglobalqualityofthesolutionand

2. asaguidetothesearchforaminimalsolution.

Correctsolutionisembeddedastheminimum.11

SegmentationasanEnergyMinimizationProblem

• Edata assignsnon-negativepenaltiestoapixellocation pwhenassigningalabeltothislocation.

• Esmooth assignsnon-negativepenaltiesbycomparingtheassignedlabelsfp andfq atadjacentpositionsp andq.

Thisoptimizationmodelischaracterizedbylocalinteractionsalongedgesbetweenadjacentpixels,andoftencalledMRF(MarkovRandomField)model.

EnergyFunction-Details

E(f) =X

[Edata

(p, fp

q2A(p)

Esmooth

• SampleDataTerm:

E(f) =X

[Edata

(p, fp

q2A(p)

Esmooth

Edata(p, fp) = (p)

(p = 0) = �logP (p 2 BG)

(p = 1) = �logP (p 2 FG)

• SampleDataTerm:

• SampleSmoothnessTerm:

E(f) =X

[Edata

(p, fp

q2A(p)

Esmooth

Edata(p, fp) = (p)

(p = 0) = �logP (p 2 BG)

(p = 1) = �logP (p 2 FG)

(p, q) = Kpq�(p 6= q) where

Kpq =exp(��(Ip � Iq)2/(2�2))

||p, q||

E(p) =X

p(p) +X

q2A(p)

pq(p, q)

p⇤ = argminp2L

• Tosolvethisproblem,transformtheenergyfunctionalintomin-cut/max-flowproblemandsolveit!

E(p) =X

p(p) +X

q2A(p)

pq(p, q)

p⇤ = argminp2L

EnergyFunction

UnaryCost(dataterm)

DiscontinuityCost

Recap:ImageasaGraph

Node/vertexEdge

GraphCutsforOptimalBoundaryDetection(BoykovICCV2001)

• Binarylabel:foregroundvs.background• Userlabelssomepixels• Exploit

– StatisticsofknownFg &Bg– Smoothnessoflabel

• Turnintodiscretegraphoptimization– Graphcut(mincut/maxflow)

Graph-Cut

EachpixelisconnectedtoitsneighborsinanundirectedgraphGoal:splitnodesintotwosetsAandBbasedonpixelvalues,andtrytoclassifyNeighborsinthesamewaytoo!

Graph-Cut

EachpixelisconnectedtoitsneighborsinanundirectedgraphGoal:splitnodesintotwosetsAandBbasedonpixelvalues,andtrytoclassifyNeighborsinthesamewaytoo!

Graph-Cut

• Eachpixel=node• AddtwonodesF&B• Labeling:linkeachpixeltoeitherForB

Desiredresult

CostFunction:Dataterm

• PutoneedgebetweeneachpixelandbothF&G• Weightofedge=minusdataterm

CostFunction:Smoothnessterm

• Addanedgebetweeneachneighborpair• Weight=smoothnessterm

Min-Cut

• Energyoptimizationequivalenttographmincut• Cut:removeedgestodisconnectFfromB• Minimum:minimizesumofcutedgeweight

Min-Cut

Source

1Graph (V, E, C)

Vertices V = {v1, v2 ... vn}Edges E = {(v1, v2) ....}Costs C = {c(1, 2) ....}

Min-Cut

Source

Whatisast-cut?

Anst-cut(S,T)divides thenodesbetweensourceandsink.

Whatisthecostofast-cut?

SumofcostofalledgesgoingfromStoT

5 + 2 + 9 = 16

Min-Cut

Whatisast-cut?

Anst-cut(S,T)divides thenodesbetweensourceandsink.

Whatisthecostofast-cut?

SumofcostofalledgesgoingfromStoT

Source

2 + 1 + 4 = 7

Whatisthest-mincut?

st-cutwiththeminimumcost

Howtocomputemin-cut?

Source

Solve the dual maximum flow problem

In every network, the maximum flow equals the cost of the st-mincut

Min-cut\Max-flow Theorem

Compute the maximum flow between Source and Sink

Constraints

Edges: Flow < Capacity

Nodes: Flow in & Flow out

Max-FlowAlgorithms

Augmenting Path Based Algorithms

1. Find path from source to sink with positive capacity

2. Push maximum possible flow through this path

3. Repeat until no path can be found

Source

Algorithms assume non-negative capacity

Flow=0

Max-FlowAlgorithms

Source

Flow=0

Max-FlowAlgorithms

Source

Flow=0+2

Max-FlowAlgorithms

Source

Flow=2

Max-FlowAlgorithms

Source

Flow=2

Max-FlowAlgorithms

Source

Flow=2

Max-FlowAlgorithms

Source

Flow=2+4

Max-FlowAlgorithms

Source

Flow=6

Max-FlowAlgorithms

Source

Flow=6

Max-FlowAlgorithms

Source

Flow=6+1

Max-FlowAlgorithms

Source

Flow=7

Max-FlowAlgorithms

Source

Flow=7

AnotherExample-MaxFlow

source

Ford & Fulkerson algorithm (1956)

Find the path from source to sink

While (path exists)

flow += maximum capacity in the path

Build the residual graph (“subtract” the flow)

Find the path in the residual graph

source

While (path exists)

source

While (path exists)

flow = 3

Lecture13:AdvanceImageSegmentation

source

While (path exists)

flow = 3

source

While (path exists)

flow = 3

source

While (path exists)

flow = 3

source

While (path exists)

flow = 6

source

While (path exists)

flow = 6

source

While (path exists)

flow = 6

source

While (path exists)

flow = 6

source

While (path exists)

flow = 11

source

While (path exists)

flow = 11

source

While (path exists)

flow = 11

source

While (path exists)

flow = 11

source

While (path exists)

flow = 13

source

While (path exists)

flow = 13

source

While (path exists)

flow = 13

source

While (path exists)

flow = 13

source

While (path exists)

flow = 15

source

While (path exists)

flow = 15

source

While (path exists)

flow = 15

source

While (path exists)

flow = 15

source

While (path exists)

flow = 15

source

Why is the solution globally optimal ?

flow = 15

source

1. Let S be the set of reachable nodes in the

residual graph

flow = 15

1. Let S be the set of reachable nodes in the

residual graph

2. The flow from S to V - S equals to the sum

of capacities from S to V – S

source

flow = 15

1. Let S be the set of reachable nodes in the residual graph

2. The flow from S to V - S equals to the sum of capacities from S to V – S

3. The flow from any A to V - A is upper bounded by the sum of capacities from A to V – A

4. The solution is globally optimal

source

2/40/2

3/30/10/1

Individual flows obtained by summing up all paths

source

cost = 18

source

cost = 23

C(x) = 5x1 + 9x2 + 4x3 + 3x3(1-x1) + 2x1(1-x3)

+ 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1)

+ 1x5(1-x1) + 6x5(1-x3) + 5x6(1-x3) + 1x3(1-x6)

+ 3x6(1-x2) + 2x4(1-x5) + 3x6(1-x5) + 6(1-x4)

+ 8(1-x5) + 5(1-x6)

source

C(x) = 2x1 + 9x2 + 4x3 + 2x1(1-x3)

+ 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1)

+ 1x5(1-x1) + 3x5(1-x3) + 5x6(1-x3) + 1x3(1-x6)

+ 3x6(1-x2) + 2x4(1-x5) + 3x6(1-x5) + 6(1-x4)

+ 5(1-x5) + 5(1-x6)

+ 3x1 + 3x3(1-x1) + 3x5(1-x3) + 3(1-x5)

source

C(x) = 2x1 + 9x2 + 4x3 + 2x1(1-x3)

+ 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1)

+ 1x5(1-x1) + 3x5(1-x3) + 5x6(1-x3) + 1x3(1-x6)

+ 3x6(1-x2) + 2x4(1-x5) + 3x6(1-x5) + 6(1-x4)

+ 5(1-x5) + 5(1-x6)

+ 3x1 + 3x3(1-x1) + 3x5(1-x3) + 3(1-x5)

3x1 + 3x3(1-x1) + 3x5(1-x3) + 3(1-x5)

3 + 3x1(1-x3) + 3x3(1-x5)

source

C(x) = 3 + 2x1 + 6x2 + 4x3 + 5x1(1-x3)

+ 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1)

+ 1x5(1-x1) + 3x5(1-x3) + 5x6(1-x3) + 1x3(1-x6)

+ 2x5(1-x4) + 6(1-x4)

+ 2(1-x5) + 5(1-x6) + 3x3(1-x5)

+ 3x2 + 3x6(1-x2) + 3x5(1-x6) + 3(1-x5)

3x2 + 3x6(1-x2) + 3x5(1-x6) + 3(1-x5)

3 + 3x2(1-x6) + 3x6(1-x5)

source

C(x) = 6 + 2x1 + 6x2 + 4x3 + 5x1(1-x3)

+ 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1)

+ 1x5(1-x1) + 3x5(1-x3) + 5x6(1-x3) + 1x3(1-x6)

+ 2x5(1-x4) + 6(1-x4) + 3x2(1-x6) + 3x6(1-x5)

+ 2(1-x5) + 5(1-x6) + 3x3(1-x5)

source

C(x) = 15 + 1x2 + 4x3 + 5x1(1-x3)

+ 3x3(1-x1) + 7x2(1-x3) + 2x1(1-x4)

+ 1x5(1-x1) + 6x3(1-x6) + 6x3(1-x5)

+ 2x5(1-x4) + 4(1-x4) + 3x2(1-x6) + 3x6(1-x5)

source

2x1 x2

x6 cost = 0

min cut

cost = 0

• All coefficients positive

• Must be global minimum

S – set of reachable nodes from s

HistoryofMax-FlowAlgorithms

Augmenting Path and Push-Relabel

n:#nodesm:#edgesU:maximumedgeweight

Algorithms assume non-negative edge

weights

ExampleSegmentationinaVideo

Image Flow GlobalOptimum

n-links st-cuts

E(x) x*

SegmentationExample

Optimality?

Seeds GrabCut(iterativeGCWithboxprior)

VascularSurfaceSegmentationviaGC10/1/15

SlideCreditsandReferences• Fredo DurandofMIT• M.Tappen ofAmazon• R.Szelisky,Univ.ofWashington/Seatle• http://www.csd.uwo.ca/faculty/yuri/Abstracts/eccv06-tutorial.html• J.Malcolm,GraphCutinTensorScale• InteractiveGraphCutsforOptimalBoundary&RegionSegmentationofObjectsin

N-Dimages.YuriBoykov andMarie-Pierre Jolly.InInternationalConference onComputerVision,(ICCV),vol.I,2001.http://www.csd.uwo.ca/~yuri/Abstracts/iccv01-abs.html

• http://www.cse.yorku.ca/~aaw/Wang/MaxFlowStart.htm• http://research.microsoft.com/vision/cambridge/i3l/segmentation/GrabCut.htm• http://www.cc.gatech.edu/cpl/projects/graphcuttextures/• AComparativeStudyofEnergyMinimizationMethodsforMarkovRandomFields.

RickSzeliski,Ramin Zabih,DanielScharstein,OlgaVeksler,VladimirKolmogorov,Aseem Agarwala,MarshallTappen,Carsten Rother.ECCV2006www.cs.cornell.edu/~rdz/Papers/SZSVKATR.pdf

• P.Kumar,OxfordUniversity.

Dr. Ulas Bagci bagci@ucfbagci/teaching/computervision16/Lec13.pdfDr. Ulas Bagci bagci@ucf.edu...

Documents