Unsupervised Learning of Categorical Segments in

Image Collections

*California Institute of Technology

**Technion

Marco Andreetto*, Lihi Zelnik-Manor**, Pietro Perona*

The Sixth IEEE Computer Society Workshop on Perceptual Organization in Computer Vision (POCV 2008)

Outline

• Motivation and related work• A probabilistic model for single image

segmentation• Unsupervised learning of categorical

segments• Experimental results• Conclusions and future works

Outline

• Motivation and related work• A probabilistic model for single image

segmentation• Unsupervised learning of categorical

segments• Experimental results• Conclusions and future works

Motivation

Motivation

Normalized cuts: Shi and Malik PAMI 2000

Motivation

Motivation

Motivation

Categorical segments: from human segmentation

Motivation

Related works

• Russell et al. CVPR 2006

• Cao and Fei-Fei ICCV 2007

• Wang and Grimson NIPS 2007

• Andreetto et al. ICCV 2007

Outline

• Motivation and related work• A probabilistic model for single image

segmentation• Unsupervised learning of categorical

segments• Experimental results• Conclusions and future works

An image as a set of segments

An image as a set of segments

K = 2N

An image as a set of segments

K = 2

1 1

2

2 1

2

Segment probabilityN

An image as a set of segments

K = 2

1 1

2 Segment probability

fk

K

Segment density

2 1

2

N

1 1

2

2 1

2

Image formation

K = 2

c

Segment probability

Label

fk

K

x

Segment density

N

What we’re looking for

Observed

Probabilistic model for clustering

c

fk

KN

x

xckcp ii ,|

fk (x)p c i k | c i

Likelihood of x to be in cluster k

Non-parametric densities

Sum of local kernels

fk 1

Ck

K x,x j x j Ck

K x,x j 1

2 j D / 2 exp x x j

2

2 j2

Outline

• Motivation and related work• A probabilistic model for single image

segmentation• Unsupervised learning of categorical

segments• Experimental results• Conclusions and future works

N

Learning categorical segments

c

fk

Kxwgk

KSegment appearance

Joint for all imagesSegment shape/color

Specific per image

M

Visual words

Filter Bank VQ

w1

w2

w3

wN

• Filter bank: 17 outputs• 256 visual words

Winn et al. ICCV 2005

…

…

Inference

N

c

fkK

xwgkK

M

)()|()|(,| cpcwpcxpwxcp

Gibbs sampling )|(),|(),|(,,| iiiiiiii ckcpcwwpcxxpwxckcp

Gibbs sampling )|(),|(),|(,,| iiiiiiii ckcpcwwpcxxpwxckcp

mkkii nckcp ,)|( Prior term:

Number of pixels in image massigned to segment k

Gibbs sampling )|(),|(),|(,,| iiiiiiii ckcpcwwpcxxpwxckcp

mkkii nckcp ,)|(

kwii incwwp ,),|(

Prior term:

Visual words term:

Number of pixels in image massigned to segment k

Number of visual word hassigned to segments k

Gibbs sampling )|(),|(),|(,,| iiiiiiii ckcpcwwpcxxpwxckcp

mkkii nckcp ,)|(

mkmk Sj

jimkSj

jimk

ii An

xxKn

cxxp,,

,,,

1),(

1),|(

Prior term:

Visual words term:

Segment term:

Number of pixels in image massigned to segment k

Number of visual word hassigned to segments k

Non-parametric densityEstimate for segment k

Affinity between observations i and j

kwii incwwp ,),|(

Outline

• Motivation and related work• A probabilistic model for single image

segmentation• Unsupervised learning of categorical

segments• Experimental results• Conclusions and future works

Experimental results (MSRC)

Classification results (MSRC)

Class Name Wang and Grimson Our model

Detection False Al. Detection False Al.

Cow 0.5662 0.0334 0.4889 0.0823

Grass N/A N/A 0.6389 0.0337

Cars 0.6838 0.2437 0.3313 0.1732

Sky N/A N/A 0.9954 0.0096

Foliage N/A N/A 0.4735 0.1122

Sea N/A N/A 0.6199 0.0174

Bikes 0.5661 0.3714 0.5436 0.0646

Faces 0.6973 0.4217 0.6161 0.0429

Running time: 18.75 sec. per image

Experimental results (Labelme)

Categorical segments (Labelme)

Segment 1: Foliage Segment 2: Buildings

Segment 1: Sky Segment 3: Street pavement

Categorical segments (scenes)

Outline

• Motivation and related work• A probabilistic model for single image

segmentation• Unsupervised learning of categorical

segments• Experimental results• Conclusions and future works

Conclusions

• We presented a model for unsupervised

learning of categorical segments

• We describe an inference method

based on Gibbs sampling

• We show some experimental results on

a standard dataset MSRC v1.

Future work

• Faster inference method (variational approximation)

• Automatic inference of the number of segments

• Learning geometric relationships between segments

Thank You