Image Segmentation with a Bounding Box Prior

Victor Lempitsky, Pushmeet Kohli, Carsten Rother, Toby SharpMicrosoft Research Cambridge

Dylan Rhodes and Jasper Lin

1

Presentation Overview

● Segmentation problem description● Background and Previous Work● Problems and Proposed Solutions

○ Formalizing tightness○ Defining tractable optimization problem for

segmentation○ Discretizing continuous approximation of solution

● Experiments and Results

2

Presentation Overview

● Segmentation problem description● Background and Previous Work● Problems and Proposed Solutions

○ Formalizing tightness○ Defining tractable optimization problem for

segmentation○ Discretizing continuous approximation of solution

● Experiments and Results

3

Segmentation Problem

How does one separate the foreground from the background with minimal user input?

4

Bounding Box

● Allows the algorithm to focus on subimage● Desired segmentation is close to sides of bounding

box

5

Bounding Box

● Allows the algorithm to focus on subimage● Desired segmentation is close to sides of bounding

box

6

Presentation Overview

● Segmentation problem description● Background and Previous Work● Problems and Proposed Solutions

○ Formalizing tightness○ Defining tractable optimization problem for

segmentation○ Discretizing continuous approximation of solution

● Experiments and Results

7

Basic Formulation

8

Basic Formulation

B is the set of pixels within the bounding box

9

Basic Formulation

E is the set of adjacent pixels within the bounding box

10

Basic Formulation

p and q are pixel indices

11

Basic Formulation

x_p can take a label of 1 for foreground or 0 for background

12

Basic Formulation

Unary potentials encode preference for foreground or background

13

Basic Formulation

Pairwise potentials enforce smoothness of the solution

14

Related Work

● Nowozin and Lampert derived framework for segmentation under connectivity constraint

● Relax NP-hard integer problem and solve resulting LP

15

Nowozin and Lampert

16

Nowozin and Lampert

17

Nowozin and Lampert

18

Presentation Overview

● Segmentation problem description● Background and Previous Work● Problems and Proposed Solutions

○ Formalizing tightness○ Defining tractable optimization problem for

segmentation○ Discretizing continuous approximation of solution

● Experiments and Results

19

Presentation Overview

● Segmentation problem description● Background and Previous Work● Problems and Proposed Solutions

○ Formalizing tightness○ Defining tractable optimization problem for

segmentation○ Discretizing continuous approximation of solution

● Experiments and Results

20

Why tightness?

21

Tightness Definition

22

Corollary

A shape x is strongly tight if and only if its intersection with the middle box has a connected component touching all four sides of the middle box

23

Energy Minimization Problem

24

Presentation Overview

● Segmentation problem description● Background and Previous Work● Problems and Proposed Solutions

○ Formalizing tightness○ Defining tractable optimization problem for

segmentation○ Discretizing continuous approximation of solution

● Experiments and Results

25

Energy Minimization Problem

26

Energy Minimization Problem

27

Convex Continuous Relaxation

28

Convex Continuous Relaxation

29

Continuous Optimization

30

Continuous Optimization

31

Additional Approximation

Intuition: Solve LP with a subset Γ' of the constraints in 3c activated

32

Calculating Γ'

1. Begin with Γ' = ∅2. Solve the LP3. Pick a group of crossing paths from Γ \ Γ' which are

violated by more than a small tolerance 4. Add these paths to Γ'5. Repeat steps 2 through 4 until all paths in Γ are

satisfied within the tolerance

33

Final Form

34

Presentation Overview

● Segmentation problem description● Background and Previous Work● Problems and Proposed Solutions

○ Formalizing tightness○ Defining tractable optimization problem for

segmentation○ Discretizing continuous approximation of solution

● Experiments and Results

35

Pinpointing Algorithm

Normally, output of LP is rounded to integer solution

36

Pinpointing Algorithm

● Pinpoint set Π contains pixels hard-assigned to foreground

37

Pinpoint Algorithm

38

Pinpoint Algorithm

39

Challenges

● Existing methods perform energy-driven shrinking over bounding box○ No guarantees optimization won’t shrink excessively

○ Stuck at poor local minima

○ Discretization of approximate solution is noisy

40

Paper’s Contributions

● Common methods initialize foreground region and perform energy-driven shrinking○ No guarantees optimization won’t shrink excessively

Solution: new tightness prior○ Stuck at poor local minima

○ Discretization of approximate solution is noisy

41

Paper’s Contributions

● Common methods initialize foreground region and perform energy-driven shrinking○ No guarantees optimization won’t shrink excessively

Solution: new tightness prior○ Stuck at poor local minima

Solution: new approximation strategies○ Discretization of approximate solution is noisy

42

Paper’s Contributions

● Common methods initialize foreground region and perform energy-driven shrinking○ No guarantees optimization won’t shrink excessively

Solution: new tightness prior○ Stuck at poor local minima

Solution: new approximation strategy○ Discretization of approximate solution is noisy

Solution: new pinpointing algorithm

43

Presentation Overview

● Segmentation problem description● Background and Previous Work● Problems and Proposed Solutions

○ Formalizing tightness○ Defining tractable optimization problem for

segmentation○ Discretizing continuous approximation of solution

● Experiments and Results

44

Experiments

● Evaluated over 50 image GrabCut dataset○ Each image comes with bounding box

● Comparison with competing methods and initialization strategies

45

GrabCut Dataset

● 50 natural images with bounding box annotations○ Includes background, outside strip, and foreground

bounding boxes

46

GrabCut Dataset

● 50 natural images with bounding box annotations○ Includes background, outside strip, and foreground

bounding boxes

47

Unary and Pairwise Terms

● Pairwise terms over 8-connected edge set

48

Relative Performance

Error rate - mislabeled pixels inside bounding boxOptimum Rank - average rank of energy of final integer program solutions

49

Relative Performance

50

Iterative Process

● Compare the following algorithms on the segmentation task:○ GrabCut with standard graph cut minimization for

all segmentation steps○ GrabCut which enforces the tightness prior for all

segmentation steps● 5 iterations each

51

Initialization Strategies

● Compare two methods for initializing foreground/background GMMs:○ InitThirds = same as Experiment 1 (outside strip +

best matches vs. poor matches) ○ InitFullBox which sets background GMM to

outside strip and foreground to whole interior of bounding box

52

Iterative Process

53

Effect of Margin Thickness

Error rates as function of margin thickness

54

Strong vs. Weak Tightness

● Strong and weak tightness lead to similar error rates in general○ same error rate (3.7%) for best model (GrabCut-

Pinpoint/InitThirds)

55

Iterative Process Comparisons

56

Conclusions

● New bounding-box based prior for interactive image segmentation

57

Conclusions

● New bounding-box based prior for interactive image segmentation

● Demonstrated segmentation tasks under this prior can be formulated as integer programs

58

Conclusions

● New bounding-box based prior for interactive image segmentation

● Demonstrated segmentation tasks under this prior can be formulated as integer programs

● Developed new optimization approaches for approximate solution of these NP-hard problems○ Can be applied to other computer vision problems e.g.

other image segmentation or silhouettes in multi-view stereo

59

Thank you!

60