Learning Decompositional Learning Decompositional Shape Models from Shape Models from

ExamplesExamples

Alex LevinshteinAlex Levinshtein

Cristian SminchisescuCristian Sminchisescu

Sven DickinsonSven Dickinson

University of TorontoUniversity of Toronto

Hierarchical ModelsHierarchical ModelsManually built hierarchical model proposed by Marr And Nishihara(“Representation and recognition of the spatial organization of three dimensional shapes”, Proc. of Royal Soc. of London, 1978)

Our goalOur goal

Automatically construct a generic hierarchical shape model from exemplars

Challenges:

Cannot assume similar appearance among different exemplars

Generic features are highly ambiguous

Generic features may not be in one-to-one correspondence

Automatically constructed Automatically constructed Hierarchical ModelsHierarchical Models

Input:

Question: What is it?

Output:

Stages of the systemStages of the systemExemplar images

Extract Blob Graphs

Match Blob Graphs (many-

to-many)

Extract PartsExtract

Decomposition Relations

Extract Attachment Relations

Assemble Final Model

Blob graphs

Many-to-many correspondences

Model parts

Model decomposition relations

Model attachment relations

Blob Graph ConstructionBlob Graph ConstructionExemplar images

Extract Blob Graphs

Match Blob Graphs (many-

to-many)

Extract PartsExtract

Decomposition Relations

Extract Attachment Relations

Assemble Final Model

Blob graphs

Many-to-many correspondences

Model parts

Model decomposition relations

Model attachment relations

Blob Graph ConstructionBlob Graph Construction

Edges are invariant to articulation

Choose the largest connected component.

On the Representation and Matching of Qualitative Shape at Multiple Scales

A. Shokoufandeh, S. Dickinson, C. Jonsson, L. Bretzner, and T. Lindeberg,ECCV 2002

Feature matchingFeature matchingExemplar images

Extract Blob Graphs

Match Blob Graphs (many-

to-many)

Extract PartsExtract

Decomposition Relations

Extract Attachment Relations

Assemble Final Model

Blob graphs

Many-to-many correspondences

Model parts

Model decomposition relations

Model attachment relations

Feature matchingFeature matchingOne-to-one matching. Rely on shape and context, not appearance!

?

Many-to-many matching

Feature embedding and Feature embedding and EMDEMD

Spectral embedding

Returning to our set of Returning to our set of inputsinputs

Many-to-many matching of every pair of exemplars.

Part ExtractionPart ExtractionExemplar images

Extract Blob Graphs

Match Blob Graphs (many-

to-many)

Extract PartsExtract

Decomposition Relations

Extract Attachment Relations

Assemble Final Model

Blob graphs

Many-to-many correspondences

Model parts

Model decomposition relations

Model attachment relations

Results of the part extraction Results of the part extraction stagestage

What is next?What is next?

Extracting attachment Extracting attachment relationsrelations

Exemplar images

Extract Blob Graphs

Match Blob Graphs (many-

to-many)

Extract PartsExtract

Decomposition Relations

Extract Attachment Relations

Assemble Final Model

Blob graphs

Many-to-many correspondences

Model parts

Model decomposition relations

Model attachment relations

Extracting attachment Extracting attachment relationsrelations

Right arm is typically connected to torso in exemplar images !

Extracting decomposition Extracting decomposition relationsrelations

Exemplar images

Extract Blob Graphs

Match Blob Graphs (many-

to-many)

Extract PartsExtract

Decomposition Relations

Extract Attachment Relations

Assemble Final Model

Blob graphs

Many-to-many correspondences

Model parts

Model decomposition relations

Model attachment relations

Extracting decomposition Extracting decomposition relationsrelations

Model construction stage Model construction stage summarysummary

Model Construction:

Clustering blobs into parts based on one-to-one matching results.

Recovering relations between parts based on individual matching and attachment results.

Assemble Final ModelAssemble Final ModelExemplar images

Extract Blob Graphs

Match Blob Graphs (many-

to-many)

Extract PartsExtract

Decomposition Relations

Extract Attachment Relations

Assemble Final Model

Blob graphs

Many-to-many correspondences

Model parts

Model decomposition relations

Model attachment relations

ConclusionsConclusions General framework for constructing a General framework for constructing a

generic decompositional model from generic decompositional model from different exemplars with dissimilar different exemplars with dissimilar appearance.appearance.

Recovering decompositional relations Recovering decompositional relations requires solving the difficult many-to-requires solving the difficult many-to-many graph matching problem.many graph matching problem.

Preliminary results indicate good model Preliminary results indicate good model recovery from noisy features.recovery from noisy features.

Future workFuture work

Construct models for objects other than Construct models for objects other than humans.humans.

Provide scale invariance during matching.Provide scale invariance during matching.

Automatically learn perceptual grouping Automatically learn perceptual grouping relations from labeled examples.relations from labeled examples.

Develop indexing and matching framework Develop indexing and matching framework for decompositional models.for decompositional models.