1Copyright 2008, Toshiba Corporation.

Björn Stenger

28 Sep 2009

2009 京都

Tutorial – Part 3

Tracking Using Classificationand Online Learning

2

Roadmap

Tracking by classificationTracking by classification

On-line BoostingOn-line BoostingMultiple Instance LearningMultiple Instance Learning

Multi-Classifier Boosting

Online Feature SelectionOnline Feature Selection

Adaptive TreesAdaptive Trees

Ensemble TrackingEnsemble Tracking

Online Random ForestOnline Random Forest

Combining off-line & on-lineCombining off-line & on-line

Tracking by optimizationTracking by optimization

3

Tracking by OptimizationExample: Mean shift tracking

Given: target location in frame t, color distribution

In frame t+1:Minimize distance

p candidate distributionq target distributiony location

Mean shift: iterative optimizationFinds local optimum

Extension: downweight by background

[Comaniciu et al. 00]

4

Support Vector Tracking [Avidan 01]

Combines SVM classifier with optical flow-based tracking

Input:- Initial guess x of object location in frame t- SVM classifier (trained on ~10,000 example images)

Algorithm:Maximize SVM classification score

SVM eqn Motion eqn (1st order Taylor) Results in this task:

Use first order Taylor approximation, obtain linear system

Prior knowledge of classifier is used in tracking process, no online update!

7

Online Selection of Discriminative Features [Collins et al. 03]

Select features that best discriminate between object and background

Feature pool:

Discriminative score: measure separability (variance ratio) of fg/bg

Within class variance should be small

Total variance should be large

8

On-line Feature Selection (2)

Input image:Feature ranking according to variance ratio

[Collins et al. 03]

Mean shift

Mean shift

Mean shift

Median

New location

Combining estimates

9

Ensemble Tracking [Avidan 05]

Use classifiers to distinguish object from background

Image Feature space

foregroundbackground

First location is provided manuallyAll pixels are training datalabeled {+1,-1}

11-dimensional feature vector

8 orientation histogram of 5x5 nhood3 RGB values

10

Ensemble Tracking [Avidan 05]

Confidence map

Train T (=5) weak linear classifiers h:

Combine into strongClassifier with AdaBoost

Build confidence map from classifier margins Scale positive margin to [0,1]

Mean shift

Find the mode using mean shift

Feature space

foreground

background

11

Ensemble Tracking Update [Avidan 05]

Test examples xi using strong classifier H(x)

For each new frame Ij

Run mean shift on confidence map

Obtain new pixel labels y

Keep K (=4) best (lowest error) weak classifiersUpdate their weights

Train T-K (=1) new weak classifiers

h1 h2 h3 h4 h5

13

AdaBoost (recap) [Freund, Schapire 97]

Input

- Set of labeled training samples

- Weight distribution over samples

for n=1 to N // number of weak classifiers

- train a weak classifier using samples and weight distribution

- calculate error- calculate classifier weight

- update sample weights

end

Algorithm

Result

[slide credit H. Grabner]

Feature space

15

From Off-line to On-line Boosting [Oza, Russel 01]

Input Input- set of labeled training samples

- weight distribution over samples

- ONE labeled training sample

- strong classifier to update

- initial sample importance

For n=1 to N- train a weak classifier using samples and weight distribution

- calculate error

- calculate confidence

- update weight distribution

End

For n=1 to N

- update weak classifier using samples and importance

- update error estimate

- update confidence

- update importance

End

Algorithm Algorithm

Off-line On-line

[slide credit H. Grabner]

16

Online Boosting [Oza, Russell 01]

Input

- ONE labeled training sample

- strong classifier

for n=1 to N // number of weak classifiers

- update weak classifier using sample and importance

- update error estimate- update classifier weight

- update sample importance

end

Algorithm

Result

Feature space

- initial sample importance

[slide credit H. Grabner]

19

Priming can help [Oza 01]

Batch learning on first 200 points, then online

20

Online Boosting for Feature Selection [Grabner, Bischof 06]

Each feature corresponds to a weak classifier Combination of simple features

21

Selectors [Grabner, Bischof 06]

A selector chooses one feature/classifier from pool.

Selectors can be seen as classifiersClassifier pool

Idea: Perform boosting on selectors, not the features directly.

22

Online Feature Selection

one sample

Init importance

Estimate errors

Select best weak classifier

Update weight

Estimate importance

Current strong classifier

For each training sample

[Grabner, Bischof 06]

Global classifier pool

Estimate errors

Select best weak classifier

Update weight

Estimate errors

Select best weak classifier

Update weight

Estimate importance

23

Tracking Principle [Grabner, Bischof 06]

[slide credit H. Grabner]

24

Adaptive Tracking [Grabner, Bischof 06]

25

Limitations [Grabner, Bischof 06]

26

Multiple Instance Learning (MIL)

Precisely labeled data is expensiveWeakly labeled data is easier to collect

Algorithm for allowing ambiguity in training data:Get bag of (data, label) pairs

Bag is positive if one or more of its members is positive.

[Keeler et al. 90, Dietterich et al. 97, Viola et al. 05]

27

Multiple Instance Learning

Classifier

Supervised learning training input

MILClassifier

MIL training input

[Babenko et al. 09]

28

Online MIL Boost [Babenko et al. 09]

At time t get more training data1 Update all candidate classifiers2 Pick best K in a greedy fashion

pool of weak classifier candidates

29

Online MIL Boost

Frame t Frame t+1

Get data (bags)

Update all classifiersin pool

Greedily add best K tostrong classifier

[Babenko et al. 09]

30

Tracking Results [Babenko et al. 09]

31

On-line / Off-line Spectrum

Tracking

DetectionGeneral object/any background detector Fixed training set

Object/Background classifier On-line update

Adaptive detector

Tracking with prior

c/f Template Update Problem [Matthews et al. 04]

Example strategies: Run detector in tandem to verify [Williams et al. 03] Include generative model [Woodley et al. 06][Grabner et al. 07] Integrate tracker and detector [Okuma et al. 04][Li et al. 07]

32

Semi-supervised

Use labeled data as priorEstimate labels & sample importance for unlabeled data

[Grabner et al. 08]

33

Tracking Results [Grabner et al. 08]

35

Beyond Semi-Supervised [Stalder et al. 09]

RecognizerObject specific“Adaptive prior”Updated by:pos: Tracked samples validated by detectorneg: Background during detection

“too inflexible”

36

Results [Stalder et al. 09]

38

Task: Tracking a Fist

39

Learning to Track with Multiple Observers

Observation Models

Off-line trainingof observer combinations

Optimal trackerfor task at hand

Labeled TrainingData

Idea: Learn optimal combination of observers (trackers) in an off-line training stage. Each tracker can be fixed or adaptive.

Given: labeled training data, object detector

[Stenger et al. 09]

40

Input: Set of observers

Single template

[NCC]

[SAD]

Normalized cross-correlation

Sum of absolute differences

Local features

[BOF]

[KLT]

[FF]

[RT]

Block-based optical flow

Kanade-Lucas-Tomasi

Flocks of features

Randomized templates

Histogram [MS]

[C]

[M]

[CM]

Color-based mean shift

Color probability

Motion probability

Color and motion probability

On-line classifiers

[OB]

[LDA]

[BLDA]

[OFS]

On-line boosting

Linear Discriminant Analysis (LDA)

Boosted LDA

On-line feature selection

Each returns a location estimate & confidence value

[Stenger et al. 09]

41

Combination Schemes

Find good combinations of observers automatically by evaluating all pairs/triplets (using 2 different schemes).

1)

2)

[Stenger et al. 09]

42

How to Measure Performance?

Run each tracker on all frames (don’t stop after first failure)

Measure position errorLoss of track when error above thresholdRe-init with detector

[Stenger et al. 09]

44

Results on Hand Data

Single observers

Pairs of observers

[Stenger et al. 09]

45

Tracking Results [Stenger et al. 09]

46

Face Tracking Results [Stenger et al. 09]

47

Multi-Classifier Boosting

Simultaneously learn image clusters and classifiers

[Kim et al. 09]

AdaBoost Multi-class boosting with gating function

48

Online Multi-Class Boosting [Kim et al. 09]

Handles multiple poses: take maximum classifier response

49

And now

TreesTrees

50

Online Adaptive Decision Trees [Basak 04]

Sigmoidal soft partitioning function at each node

hyperplane

Activation value at node i

Complete binary trees, tree structure is maintained, each class = subset of leaves, label leaf nodes beforehand

For each training sample, adapt decision hyperplanes at all inner nodes via gradient descent on error measure (leaf node activation)

51

Adaptive Vocabulary Forests [Yeh et al. 07]

Application: Efficient indexing, leafs represent visual words Batch learning: hierarchical k-means, cf. [Nister and Stewenius 06]

[slide credit T. Yeh]

52

Incremental Building of Vocabulary Tree [Yeh et al. 07]

53

Tree Growing by Splitting Leaf Nodes [Yeh et al. 07]

54

Tree Adaptation with Re-Clustering [Yeh et al. 07]

Identify affected neighborhood

Remove exisiting boundaries

Re-Cluster points

55

Accuracy drops when Adaptation is stopped [Yeh et al. 07]

Recent accuracy

T=100

R(j) = 1

if top ranked retrieved image belongs to same group

56

[Yeh et al. 07]Tree Pruning

Limit the number of leaf nodes

Keep record of inactivity period at each node, if limit reached, remove nodes with least-recently used access

Allows for restructuring of heavily populated areas

57

On-line Random Forests [Saffari et al. 09]

…

For each tree t

Input: New training example

Update tree t with k times

Estimate Out-of-bag error

end

P(Discard tree t and insert new one) =

Random forest

58

Leaf Update and Split [Saffari et al. 09]

Set of random split functions

Split node when1) Number of samples in node > threshold 12) Gain of best split > threshold 2

class k

Compute gain of each potential split function

Current leaf node

59

Results [Saffari et al. 09]

Convergence of on-line RF classification to batch solution on USPS data set

Tracking error of online RF compared to online boosting

60

Conclusions

On-line versions exist for Boosting and Random ForestsOn-line versions exist for Boosting and Random Forests

Experimentally good convergence results (but few Experimentally good convergence results (but few theoretical guarantees)theoretical guarantees)

Useful for Tracking via ClassificationUseful for Tracking via Classification

A lot of code has been made available online by authorsA lot of code has been made available online by authors

Detection – Tracking SpectrumDetection – Tracking Spectrum

Adaptation vs. DriftAdaptation vs. Drift

61

ReferencesAvidan, S.,Support Vector Tracking, IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), Hawaii, 2001.

Avidan, S.,Support Vector Tracking ,IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), Vol. 26(8), pp. 1064--1072, 2004.

Avidan, S.,Ensemble Tracking,IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), Vol. 29(2), pp 261-271, 2007.

Avidan, S.,Ensemble Tracking,IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), San Diego, USA, 2005.

Babenko, B., Yang, M.-H., Belongie, S.,Visual Tracking with Online Multiple Instance Learning,Proc. CVPR 2009.

Basak, J.,Online adaptive decision trees, Neural Computation, v.16 n.9, p.1959-1981, September 2004.

Collins, R. T., Liu, Y., Leordeanu, M.,On-Line Selection of Discriminative Tracking Features,IEEE Transaction on Pattern Analysis and Machine Intelligence (PAMI), Vol 27(10), October 2005, pp.1631-1643.

Collins, R. T., Liu, Y.,On-Line Selection of Discriminative Tracking Features,Proceedings of the 2003 International Conference of Computer Vision (ICCV '03), October, 2003, pp. 346 - 352.

Comaniciu, D., Ramesh, V., Meer, P.,Kernel-Based Object Tracking, IEEE Trans. Pattern Analysis Machine Intell., Vol. 25, No. 5, 564-575, 2003.

Comaniciu, D., Ramesh, V., Meer P.,Real-Time Tracking of Non-Rigid Objects using Mean Shift, IEEE Conf. Computer Vision and Pattern Recognition, Hilton Head Island, South Carolina, Vol. 2, 142-149, 2000.

T. G. Dietterich and R. H. Lathrop and T. Lozano-Perez,Solving the multiple instance problem with axis-parallel rectangles.Artificial Intelligence 89 31-71, 1997.

Freund, Y. , Schapire, R. E. ,A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, 55(1):119–139, August 1997.

H. Grabner, C. Leistner, and H. Bischof,Semi-supervised On-line Boosting for Robust Tracking.In Proceedings European Conference on Computer Vision (ECCV), 2008.

H. Grabner, P. M. Roth, H. Bischof,Eigenboosting: Combining Discriminative and Generative Information, IEEE Conference on Computer Vision and Pattern Recognition, 2007.

H. Grabner, M. Grabner, and H. Bischof,Real-time Tracking via On-line Boosting,In Proceedings British Machine Vision Conference (BMVC), volume 1, pages 47-56, 2006.

H. Grabner, and H. Bischof,On-line Boosting and Vision,In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), volume 1, pages 260-267, 2006.

J. D. Keeler , D. E. Rumelhart , W.-K. Leow, Integrated segmentation and recognition of hand-printed numerals, Proc. 1990 NIPS 3, p.557-563, October 1990, Denver, Colorado, USA.

T.-K. Kim and R. Cipolla, MCBoost: Multiple Classifier Boosting for Perceptual Co-clustering of Images and Visual Features, In Advances in Neural Information Processing Systems (NIPS), Vancouver, Canada, Dec. 2008.

T-K. Kim, T. Woodley, B. Stenger, R. Cipolla, Online Multiple Classifier Boosting for Object Tracking, CUED/F-INFENG/TR631, Department of Engineering, University of Cambridge, June 2009.

62

Y. Li, H. Ai, S. Lao, M. Kawade, Tracking in Low Frame Rate Video: A Cascade Particle Filter with Discriminative Observers of Different Lifespans, Proc. CVPR, 2007.

I. Matthews, T. Ishikawa, and S. Baker,The template update problem. In Proc. BMVC, 2003

I. Matthews, T. Ishikawa, and S. Baker,The Template Update Problem,IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 26, No. 6, June, 2004, pp. 810 - 815.

K. Okuma, A. Taleghani, N. De Freitas, J. Little, D. G. Lowe, A Boosted Particle Filter: Multitarget Detection and Tracking ,European Conference on Computer Vision(ECCV), May 2004.

Oza, N.C.,Online Ensemble Learning, Ph.D. thesis, University of California, Berkeley.

Oza, N.C. and Russell, S., Online Bagging and Boosting.In Eighth Int. Workshop on Artificial Intelligence and Statistics, pp. 105–112, Key West, FL, USA, January 2001.

Oza, N.C. and Russell, S., Experimental Comparisons of Online and Batch Versions of Bagging and Boosting, The Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, California, 2001.

Saffari, A., Leistner C., Santner J., Godec M., Bischof H.,On-line Random Forests, 3rd IEEE ICCV Workshop on On-line Computer Vision, 2009.

S. Stalder, H. Grabner, and L. Van Gool,Beyond Semi-Supervised Tracking: Tracking Should Be as Simple as Detection, but not Simpler than Recognition.In Proceedings ICCV’09 WS on On-line Learning for Computer Vision, 2009.

B. Stenger, T. Woodley, R. Cipolla,Learning to Track With Multiple Observers.Proc. CVPR, Miami, June 2009.

References & CodeP. A. Viola and J. Platt and C. Zhang,Multiple instance boosting for object detection,Proceedings of NIPS 2005.

O. Williams, A. Blake, and R. Cipolla, Sparse Bayesian Regression for Efficient Visual Tracking, in IEEE Transactions on Pattern Analysis and Machine Intelligence, IEEE Computer Society, August 2005.

O. Williams, A. Blake, and R. Cipolla, A Sparse Probabilistic Learning Algorithm for Real-Time Tracking,in Proceedings of the Ninth IEEE International Conference on Computer Vision, October 2003.

T. Woodley, B. Stenger, R. Cipolla,Tracking Using Online Feature Selection and a Local Generative Model,Proc. BMVC, Warwick, September 2007.

T. Yeh, J. Lee, and T. Darrell,Adaptive Vocabulary Forests for Dynamic Indexing and Category Learning.Proc. ICCV 2007.

Code:Severin Stalder, Helmut GrabnerOnline Boosting, Semi-supervised Online Boosting, Beyond Semi-Supervised Online Boostinghttp://www.vision.ee.ethz.ch/boostingTrackers/index.htm

Boris BabenkoMIL Trackhttp://vision.ucsd.edu/~bbabenko/project_miltrack.shtml

Amir Saffarihttp://www.ymer.org/amir/software/online-random-forests/