Local Affine Feature Tracking in Films/Sitcoms

Chunhui GuCS 294-6

Final PresentationDec. 13, 2006

Objective

• Automatically detect and track local affine features in film/sitcom frame sequences.

– Current Dataset: Sex and the City– Why sitcom?

• Simple daily environment• Few or no special effects• Repeated scenes

Outline

• Preprocessing• Tracking Algorithm

– Pairwise local matching– Robust features

• Feature Matching across Shots• Results

– Feature matching vs baseline color histogram– Time complexity– When does tracking fail

Preprocessing

FrameExtraction

(i-1)’th shot i’th shot

ShotDetection

MSER Interest PointDetection

SIFT FeatureExtraction

Tracking Algorithm

• Basic: Pairwise Matching

Frame i Frame j=i+1

imf

,i im mx y

jnf

Tracking Algorithm

• Basic: Pairwise Matching

Frame i Frame j=i+1

imf

,i im mx y

jnf

Tracking Algorithm

• Basic: Pairwise Matching

Frame i Frame j=i+1

imf

,i im mx y

jnf min fd

Thresholding on both minimum distance and ratio

Tracking Algorithm

• Basic: Pairwise Matching

Frame i Frame j=i+1

imf

,i im mx y

jnf

Tracking Algorithm

• Basic: Pairwise Matching

Frame i Frame j=i+1

imf

,i im mx y

jnf

Tracking Algorithm

• Problem of Pairwise Matching– Sensitive to occlusion and feature misdetection

• Solutions:– Use multiple overlapping windows– Backward Matching

• Match features in current frame to features in all previous frames within the shot

• Pruning process (reduce computation time)

• Select a proportion of features that have longer tracking length as robust features

Shot grouping/Scene Retrieval

60

601 2, ,...rf mf x x x

56

561 2, ,...rf mf x x x

10746 10747 10772

Shot 49

10933 10934 10968

Shot 53

11393 11394 11435

Shot 56

Shot 60

11533 11534 11560

Scene 5

49

491 2, ,...rf mf x x x

53

531 2, ,...rf mf x x x

Inter-Shot Matching

Shot I Shot J

1 11 2, ,...I mf x x x

2 21 2, ,...I mf x x x

1 11 2, ,...J nf x x x

2 21 2, ,...J nf x x x

1 2, ,...q qJ nf x x x 1 2, ,...

p pI mf x x x

D

“Confusion Table”

Ground Truth50 55 60 65 70 75

50

55

60

65

70

75

Color Histograms50 55 60 65 70 75

50

55

60

65

70

75

Feature Matching50 55 60 65 70 75

50

55

60

65

70

75

ROC

0 0.2 0.4 0.6 0.8 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

False Alarm

Tru

e D

etec

tion

ROC curve of Feature Matching

When Does Tracking Fail?• Tracking feature outside local window

– Rare when continuous tracking– Happens when occlusion occurs

• Same feature splitting to two or more groups– Long occlusion– Multiple matching in a single frame

Frame i Frame j=i+1

imf

,i im mx y

jnf

Computation Complexity• Everything except for MSER and SIFT algorithms are

implemented in Matlab (slow…)

Complexity Time

Frame Extraction O(N) ~0.3s/frame

Shot Detection O(N*f(B)) ~0.07s/frame (B=16)

MSER Detection O(N) ~0.3s/frame

SIFT Detection O(N) ~0.9s/frame

Feature Tracking O(N*F*W*L) ~0.5s/frame

Matching across shots

O(S2*T2) ~1s/shot pair

N: # of frames; (30,000) B: # of bins for color hist (16) F: ave. # of features per frame; (400) W: Local window size; (15)L: tracking length; (20) T: ave. # of robust trackers per shot; (300)S: # of shots; (35)

Conclusion

• We successfully implemented local affine feature tracking in sitcom “sex and the city”. The tracking method is robust to occlusion and feature misdetection.

• Although no quantitative precision/recall curve (hard to find ground truth), the demonstration shows that precision is almost perfect with good recall performance.

• We show one successful application of using robust features to associate similar shots together for scene retrieval.

Future Work

• Implement algorithm in real-time (C/C++)

• Search unique shots in films/sitcoms

• Separate indoor scenes from outdoor scenes

• Determine context of the scene

Acknowledgement