Spatial Histograms for Head Tracking

Sriram RangarajanDepartment of Electrical and Computer Engineering,Clemson University, Clemson, SC 29634

Overview of trackerIntensity Gradients (works on the boundary of the ellipse)

Modules that are complementary to Modules that are complementary to gradients :gradients :

1.1. Color histogramsColor histograms

2.2. SpatiogramsSpatiograms

3.3. Co-occurrence matricesCo-occurrence matrices

4.4. Log-Gabor histogramsLog-Gabor histograms

5.5. Haar histogramsHaar histograms

6.6. Edge-orientation histogramsEdge-orientation histograms

Complementary module (works inside the ellipse)

Gradient module

N

iNg ii1

1 |)()(|)( sgns

Normal to points on ellipse Gradient score

Likelihood score

[Stan Birchfield, 1998]

Overview of modules used

ModelModel

histogramhistogram

(from first(from first

frame)frame)

TargetTarget

histogramhistogram

(from(from

current current frame)frame)

Similarity measureSimilarity measure

Likelihood score from moduleLikelihood score from module

Convert to percentage score, combine with intensity gradient module likelihood and update “state”.

Similarity measure between model and target histograms

)(min)(max

)(min)(

)(icSsicSs

icSsc

ii

im ss

ss

s

Ni

Ni

iI

iMiIm

1

1

)(

))(),(min()(

s

ssHistogram intersection [Swain & Ballard1991]

Likelihood normalization

Overview of modules

Color histograms

Only color information (no spatial information)

Spatiograms Color information + limited spatial information ( global)

Edge-orientation histograms

Only spatial information

Co-occurrence matrices

Color information + limited spatial information ( local)

Log-Gabor histograms

Only spatial information (no color)

Haar histograms Only spatial information (no color)

Color Histograms

Ignore spatial information (most cases) Computationally efficient, simple,

robust and invariant to any one-to-one spatial transformations

Computing color histograms

Index for color channelPixels in a bin

)max(Cnbins

1 11

y)(x,Ci Single color channel of image

Number of bins for channel C1

Spatiograms

Higher-order histograms that capture spatial information globally

Captures both values of pixels and a limited amount of their spatial relationship

Bins are weighted by mean and covariance of pixels contributing to it

[Birchfield and Rangarajan, CVPR 2005]

Spatiograms and histograms

A histogram(no spatial information)

A spatiogram(some spatial Information)

Σ

µ

A histogram(no spatial information)

A spatiogram(some spatial Information)

Σ

µ

A histogram(no spatial information)

A spatiogram(some spatial Information)

Σ

µ

An illustrative example

Three poses of a head

Image generated from histogram

Image generated from spatiogram

Co-occurrence matrices

Used for texture analysis Captures the local spatial relationships between

colors (or gray levels) Normally used for gray-level images

No. of pixel pairs with value (x,y)

Co-occurrence matrices

10 11

10 13

10

10

13 10 11

13 10 11

10

11

1310 11

13

Image Co-occurrence matrix

Local spatial relationships

Color values (C)

(C)

Texture histograms

*Filter bank

Image

=Histogram

(Haar Waveletsor

Log-Gabor filters)

Haar histograms

Histogram of image after convolving with 3-level Haar pyramid:

Haar histogram (at scale S and orientation O.)

Image obtained by convolving with Haar pyramid at scale S and orientation O

Log-Gabor histograms

Similar to Haar histograms, but uses a bank of log-Gabor filters.

Log-Gabor histogram Image obtained by convolving with filter bank at scale S and orientation O

Edge-orientation histograms

Obtained from gradient information

Complete reliance on spatial information

Histogram bin is decided by orientation of a pixel

Computing edge-orientation histograms

* =Difference of

Gaussian kernel (DoG)

ImageEdge-orientation

Histogram

Edge-orientation histograms

Computed from gradient images obtained by convolving image with Difference of Gaussian (DoG) kernel in x and y

Orientation for pixel along vertical direction is 0

Results: log-Gabor histograms

log-Gaborhistogram

colorhistogram

Legend:

Results: Haar histograms

Haarhistogram

colorhistogram

Legend:

Results: Edge-orientation histograms

Edge-orientationhistogram

colorhistogram

Legend:

Results: Spatiograms

spatiograms

colorhistogram

Legend:

Results: Co-occurrence matrices

Co-occurrencematrices

colorhistogram

Legend:

Overview of results

Color histograms

Distracted by skin-colored background

Spatiograms Tracks target in skin-colored background and clutter

Edge-orientation histograms

Fails in a cluttered background

Co-occurrence matrices

Tracks target in skin-colored background and clutter

Log-Gabor histograms

Fails in a cluttered background

Haar histograms Fails in a cluttered background

Mean errors in x and y for Sequence 1

Mean errors in x and y for Sequence 2

Conclusion

Limited amount of spatial information drastically improves tracking results

Color information also important: With only spatial information:

tracker is distracted by cluttered background With only color:

tracker is distracted by skin-colored background Global spatial information is the most effective

(spatiograms)

Thank You!