1

Visual Object Tracking

Alireza Asvadi

University of Coimbra

May 2014

Video-Vigilance and Biometrics

A review on:

2

Definition and application

“Object tracking” from different points of view

Yilmaz et al. 06

Yang et al. 11

David Forsyth & Jean Ponce 12

Outline:

3

Definition and application

4

What is object tracking?

Estimating the trajectory of an object over time by locating its position in every frame.

5

motion-based recognition

automated surveillancevideo indexing

human-computer interaction

traffic monitoringvehicle navigation…

Applications:

6

Object tracking from different points of view

Yilmaz et al. 06

Yang et al. 11

David Forsyth & Jean Ponce 12

7

How to categorize? There are different points of view

Object tracking

Yilmaz et al. 06

Yang et al. 11

David Forsyth & Jean Ponce 12

8

Object Tracking: A Survey

Object detection

Object tracking

Point detectorsBackground subtractionSegmentationSupervised learning

Point trackingKernel tracking

Silhouette tracking

Object representation

Feature selection for tracking

Yilmaz et al. 06

9

Object representation:

Points, primitive geometric shapes, object silhouette and contour, articulated shape models and skeletal models.

Object representations are chosen according to the application domain.

10

Feature selection for tracking:

Desirable property of a visual feature is its uniqueness so that the objects can be easily distinguished in the feature space.

many tracking algorithms use a combination of these features.

Color: RGB, L*u*v, L*a*b, HSV

Edges: less sensitive to illumination changes

Optical flow: displacement vectorsBrightness constraint [Horn & Schunk 81]

Texture: measure of the intensity variation of a surface

11

Object detection:

Point detectors: are used to find interest points in images which have an expressive texture in their respective localities (ex. Harris & SIFT).A desirable interest point: invariance to illumination and camera viewpoint

Background subtraction: Object detection can be achieved by building a representation of the scene called the background model and then finding deviations from the model for each incoming frame.

Segmentation: The aim of image segmentation algorithms is to partition the image into perceptually similar regions and can be used for object detection(ex. Mean shift clustering & graph cut).

Supervised learning: Object detection can be performed by learning different object views automatically from a set of examples by means of a supervised learning mechanism (ex. SVM).

Every tracking method requires an object detection mechanism either in every frame or when the object first appears in the video.

12

Taxonomy of tracking methods:

13

Point tracking: Objects detected in consecutive frames are represented by points.

Deterministic Methods: Proximity, maximum velocity (r denotes radius), small velocity-change, common motion, rigidity constraints

Statistical Methods: Kalman filter, particle filter

14

Kernel tracking: Kernel refers to the object shape and appearance. For example, the kernel can be a rectangular template or an elliptical shape with an associated histogram.

Template and density based appearance models: Ex. Template matching & mean shift tracking. Advantage of the mean shift is the elimination of a brute force search.

Multiview appearance models: The objects may appear different from different views. Different views of the object can be learned offline and used for tracking.

15

Silhouette tracking: Silhouette-based object tracker try to find the object region in each frame by means of an object model generated using the previous frames.

Shape matching: Object model is in the form of an edge map. Shape matching performed similar to tracking based on template matching.

Contour tracking: Iteratively evolve an initial contour in the previous frame to its new position in the current frame. This contour evolution requires that some part of the object in the current frame overlap with the object region in the previous frame.

16

Summary:

Point tracking: Good for finding Geometrical and 3D structure of object.Point correspondence is a complicated problem-specially in the presence of occlusions, misdetections, entries, and exits of objects.

Kernel tracking: Real time applicability.One of the limitations of primitive geometric shapes for object representation is thatparts of the objects may be left outside of the defined shape while parts of the background may reside inside it.

Silhouette tracking: Good for modeling object with complex shape.Sensitive to noise. Not capable to deal with object split and merge.

17

Recent advances and trends in visual tracking: A review

Feature descriptors for visual tracking

Online learning based tracking methods

Yang et al. 11

Generative methods

Discriminative methods

18

Feature descriptors for visual tracking:

Gradient features: statistical summarization of the gradients

Color features:

Texture features:

Spatio-temporal features:

Multiple features fusion: HOG-LBP, …

HOGmulti-resolution HOGSIFTSURF…

CSIFT (concatenation of the hue histogram with the SIFT descriptor. Invariant to light intensity)

LBP (Local Binary Patterns) a grayscale invariant textureMB-LBP (multi-scale block LBP)…

HOG/HOF (Histograms of oriented gradients & optic flow)HOG3D (histograms of 3D gradient orientations)ESURF (extends the image SURF descriptor to video)DLBP (dynamical Local Binary Patterns)…

19

Online learning based tracking methods:

The generative method builds a model to describe the appearance of an object and then finds the object by searching for the region most similar to the reference model in each frame.The object model is often updated online to adapt to appearance changes.generative methods would easily fail within cluttered background.

Generative online learning methods:

Discriminative methods pose object tracking as a binary classification problem in which the task is to determine a decision boundary that distinguishes the object from the background without the need to a complex model characterizing the object.To handle appearance changes, the classifier is updated incrementally over time.A major shortcoming of discriminative methods is their noise sensitivity.

Discriminative online learning methods:

There are methods which combine these two methods.

20

Object tracking

Computer Vision A Modern Approach 2nd Edition,Ch 11, TrackingForsyth et al. 12

Tracking by detection

Tracking with dynamics

Applying Data association

21

Time t Time t+1 Time t+n

Tracking by detection:we have a strong model of the object, we detect the object independently in each frame and can record its position over time.

Occlusions

TimeSimilar Objects

Problems:

22

Tracking with dynamics:

Key idea: Given a model of expected motion, predict where objects will occur in next frame.Filtering Problem: Estimate of c based on prediction a and measurement b

Observation (Detected object) + Dynamics

The information from the predictions and measurements are combined to provide the best possible estimate of the location of the train.

the product of two Gaussian functions is another Gaussian function.

R. Faragher , “Understanding the Basis of the Kalman Filter Via a Simple and Intuitive Derivation,” IEEE Signal Processing Magazine, September 2012.

The Kalman filter:

a bc

23

Problems: Yet it is not good enough

So far, we’ve assumed the entire measurement to be relevant to determining the state. In reality, there may be uninformative measurements may belong to different tracked objects.

Data association: Task of determining which measurements go with which tracks.

Tracking Matching: Match should be close to predicted position

Gating: Omit MeasurementsOutside the gate

24

Tracking: Detection(observation)+ dynamics +Data association

Applying Data association (Gating) Predicted Values By Kalman Filter (Green)

NCCNCC+KalmanNCC+Kalman+DA

25

Tracking by detectiondetect the object independently in each frame tracking=Detection Detection Methods:

Tracking with dynamicsincorporate object dynamics to tracking Methods:tracking=Detection(observation)+dynamics

Applying Data associationEliminate highly unlikely measurementstracking=Detection(observation)+ dynamics +Data association

Summary:

Point detectorsTemplate matchingdensity Based appearance modelsBackground Subtraction…

Filtering MethodsKalman filter…

Methods:Tracking MatchingGating…

26

Reference:

A. Yilmaz, O. Javed, and M. Shah, “Object tracking: A survey,” ACM Computing Surveys, Vol. 38, No. 4, pp. 1–45, December 2006.

H. Yang, L. Shao, F. Zheng, L. Wang, and Z. Song, “Recent advances and trends in visual tracking: A review,” Neurocomputing, Vol. 74, No. 18, pp. 3823-3831.

D. A. Forsyth, J. Ponce, “Computer Vision: A Modern Approach,” Prentice Hall,2nd Edition, 2012.

R. Faragher , “Understanding the Basis of the Kalman Filter Via a Simple and Intuitive Derivation,” IEEE Signal Processing Magazine, September 2012.

27

Thank you for your attention