ProbabilisticGraph and Hypergraph Matching

Ron Zass & Amnon Shashua

School of Engineering and Computer Science,The Hebrew University, Jerusalem

Zass & Shashua

Example: Object Matching No global affine transform.

Images from: www.operationdoubles.com/one_handed_backhand_tennis.htm

Local affine transforms + small non-rigid motion.

Match by local features + local structure.

Zass & Shashua

Hypergraph Matching inComputer Vision

Problem: Distances are not affine invariant.

In graph matching, we describe objects asgraphs (features nodes, distances edges)and match objects by matching graphs.

Zass & Shashua

Hypergraph Matching inComputer Vision

Affine invariant properties. Properties of four or more points. Example: area ratio, Area1 / Area2 Describe objects as hypergraphs

(features nodes, area ratio hyperedges)

Match objects by doing Hypergraph Matching.

In general, if n points are required to solve the local transformation, d = n+1 points are required for an invariant property.

2

13

4

Area1

Area2

Zass & Shashua

Related WorkHypergraph matching

Hypergraph matching: Wong, Lu & Rioux, PAMI 1989 Sabata & Aggarwal, CVIU 1996 Demko, GbR 1998 Bunke, Dickinson & Kraetzl, ICIAP 2005

All search for an exact matching. Edges are matched to edges of the exact same

label. Search algorithms for the largest sub-isomorphism.

We are interested in an inexact matching. Edges are matched to edges with similar labels. Find the best matching according to some score

function.

Unrealistic

Zass & Shashua

Related WorkInexact Graph Matching

Popular line of works: Continuous relaxation As an SDP problem

Schellewald & Schnörr, CVPR 2005 As a spectral decomposition problem

Leordeanu & Hebert, ICCV 2005; Cour, Srinivasan & Shi, NIPS 2006 Iterative Linear approximations using Taylor

expansions Gold & Rangarajan, CVPR 1995

And many more. Some continuous relaxation may be interpreted

as soft matching.

Our work differ:We assume probabilistic interpretation of

the input and extract probabilistic matching

Zass & Shashua

From Soft to Hard

Given the optimal soft solution X , the nearest hard matching is found by solvinga Linear Assignment Problem. The two steps (soft matching and nearest

hard matching) are optimal. The overall

hard matchingis not optimal(NP-hard).

SoftMatching

HardMatching

BetterMatching

Zass & Shashua

Hypergraph Matching

Two directed hypergraphs of degree d,G=(V,E) and G ’=(V ’,E ’). A hyper-edge is an ordered d -tuple of

vertices. Include the undirected version as a private

case. Matching: m : V V ’ Induce edge matching, m : E E ’,

m (v1 ,…,vd ) = (m (v1 ),…,m (vd ) )

Zass & Shashua

ProbabilisticHypergraph Matching

Input: Probability that an edge frome E match to an edge in e ’ E ’:

Output: Probability that two vertices match

We will derive an algebraic connection between S and X , and then use it for finding the optimal X .

',|')(Pr', GGeemS ee

',|')(Pr', GGvvmX vv

Zass & Shashua

Kronecker Product

Kronecker product between an i xj matrix A to a k xl matrix B is a ik xjl matrix:

BaBa

BaBaBA

ijj

i

1

111

didi AAA 11

AA di

d1

Zass & Shashua

S ↔ X connection

Assumption Proposition (S ↔ X connection)

Proof

',|)()( 21 GGvmvm

XS d

d

ivv

d

iii

ee

iiX

GGvvm

GGeemS

1',

1

',

',|')(Pr

',|')(Pr

',,''

,,,

1

1

d

d

vvevve

Zass & Shashua

S ↔ X connection for graphs

V ’ xV ’

V xV

XX S

V ’

V X

Xxij

XXS

',',)','(),,( 22112121 vvvvvvvv XXS

Zass & Shashua

Globally Optimal Soft Hypergraph Matching

Nearest to S , where X is a validmatrix of probabilities:

Vertex can be left unmatched. With equalities, all vertices must be

matched.

1111

T

d

X

XXXts

XSdist

,,0..

,min

Xd

1111 TXX ,

Zass & Shashua

Cour, Srinivasan & Shi 2006

Our result can explain somepreviously used heuristics.

Cour et al 2006 preprocessing:Replace S with the nearest doubly stochastic matrix (in relative entropy) before any other graph matching algorithm.

Proposition: For X ≥ 0, X is doubly stochastic iff is doubly stochastic.

is doubly stochastic.Xd

XS d

Zass & Shashua

Globally Optimal Soft Hypergraph Matching

We use the Relative Entropy (Maximum Likelihood) error measure,

Global Optimum, Efficient.

1111

T

d

X

XXXts

XSdist

,,0..

,min

ijijji ij

ijij BA

B

AABADBAdist

,

log)||(),(

Zass & Shashua

Globally Optimal Soft Hypergraph Matching

Define:

XSDX d

X ||minarg*

1111 XXXYDX TdT

X ||minarg*

d

iveevee

eevv

i

i

SY1

'''||

',',

Convex problem, with |V |x|V ’| inputs and outputs!

Zass & Shashua

Globally Optimal Soft Hypergraph Matching

Define , the number of matches.

1111 XXXYDX TdT

X ||minarg*

11 Xk T

kXXXX

tsXYDkX

TTX

111111 ,,,0

..||minarg*

X *

(k ) is convex in k. We give optimal solution for X

* (k ),

and solve for k numerically(convex minimization in single variable).

Zass & Shashua

Globally Optimal Soft Hypergraph Matching

Define three sub-problems ( j =1,2,3):

Each has an optimal closed form solution.

321..||min CCCXtsXYDX

kXXXC

XXXC

XXXC

T

T

11

11

11

,0|

,0|

,0|

3

2

1

)(||min)( HXtraceXYDHP T

CXj

j

Zass & Shashua

Successive Projections[Tseng 93, Censor & Reich 98]

Set For t =1,2,… till convergence:

For j = 1,2,3:

.,0 )0()0( YX jj

)(1

)1()( tj

tjj

tj XfPX

)()(1

)1()( tj

tj

tj

tj XfXf

XYDXfXX tt ||,)1(3

)(0

Optimal!

Zass & Shashua

Globally Optimal Soft Hypergraph Matching

When the hypergraphs are of the same size, and all vertices has to be matched,

our algorithm reduces to the Sinkhorn algorithm for nearest doubly stochastic matrix in relative entropy.

,111 TXX

Zass & Shashua

Sampling Given Y, the problem size reduce to |V |x|V ’|. Calculate Y : simple sum on all hyper-edges. Problem: Compute S, the hyper-edge

to hyper-edge correlation. Sampling heuristic: For each vertex, use

only z closest hyper-edges.

Heuristic applies to transformation that are locally affine (but globally not affine).

O(|V |·|V ’|·z 2) correlations.

Zass & Shashua

Runtime

Spectral MatchingLeordeanu05

Our scheme (graphs)

Our scheme (hypergraphs)

Without edge correlations time With hyperedge correlations time

(50 points)

Hyperedges per vertex

Zass & Shashua

Experiments on Graphs

to a single graph to both graphs

Spectral MatchingLeordeanu05 withCour06

preprocessing

Spectral MatchingLeordeanu05

Our scheme

Two duplicates of 25 points.

Graphs based on distances.

Additional random points.

Zass & Shashua

Experiments on Graphs

Spectral MatchingLeordeanu05 withCour06

preprocessing

Spectral MatchingLeordeanu05

Our scheme

Mean distance between neighboring points is 1.

One duplicate distorted with a random noise.

Spectral uses Frobenius norm – should have better resilience to additive noise.

Due to the global optimal solution, Relative Entropy shows comparable results.

Zass & Shashua

Limitations of Graphs

Spectral MatchingLeordeanu05 withCour06

preprocessing

Spectral MatchingLeordeanu05

Our scheme (graphs)

Our scheme (hypergraphs,z=60)

Affine Transformation (doesn’t preserve distances)

random distortion additional points to a single graph to both

graphs

Zass & Shashua

Feature Matching inComputer Vision

Based solely on local appearance: Different features might look the same. Same feature might look differently.

Describe objects by local features (e.g., SIFT).

Match objects by matching features.

Zass & Shashua

Global Affine Transformation

Images from: www.robots.ox.ac.uk/vgg/research/affine/index.html

Spectral Graph Matching Hypergraph Matching based on distances based on area ratio

10/33 mismatches no mismatches

Zass & Shashua

Non-rigid Matching Match first and last frames of a 200

frames video (6 seconds), using [Torresani & Bregler, Space-Time Tracking, 2002]

features.

Videos and points from: movement.stanford.edu/nonrig/

Zass & Shashua

Non-rigid Matching

Videos and points from: movement.stanford.edu/nonrig/

Zass & Shashua

Summary Structure translates to hypergraphs,

not graphs. Probabilistic interpretation leads to a simple

connection between input and output:

Globally Optimal solution underRelative Entropy (Maximum Likelihood).

Efficient for both graphs and hypergraphs. Apply to graph matching problems as well.

XS d