+ All Categories
Home > Documents > Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or...

Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or...

Date post: 21-Jul-2020
Category:
Upload: others
View: 4 times
Download: 0 times
Share this document with a friend
85
Moving Object Tracking Harpreet S. Sawhney [email protected] Princeton University COS 429 Lecture Dec. 6, 2007
Transcript
Page 1: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Moving Object Tracking

Harpreet S. [email protected]

Princeton UniversityCOS 429 Lecture

Dec. 6, 2007

Page 2: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Recapitulation : Last Lecture

• Moving object detection as robust regression with outlier detection

• Simultaneous multiple surface/moving object estimation

• Expectation-Maximization (EM) algorithm as a formal mechanism for multiple model estimation & its application to multiple motion detection and estimation

Page 3: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

The Tracking Problem• Maintain identity across time of designated or automatically

detected objects

Page 4: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

The Tracking Problem : Issues• Object Detection : Designated or automatically detected

• Object State Instantiation: Object representation– Position, Velocity, Shape, Color, Appearance, Template…

• State Prediction: – Having seen what state do these measurements predict for the

next time instant i?– Need a representation for

• Data Association:– Which of the measurements at the i-th instant correspond to the predicted

state at that instant ?– Use to establish the correspondence

• State Update:– With the corresponding measurement established for instant i, compute

an estimate of the optimal new state through

}y,,y,y{ 1-i10 K

)yY,,yY|X(P 1-i1-i00i == K

)yY,,yY|X(P 1-i1-i00i == K

iy)yY,,yY|X(P ii00i == K

Page 5: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Object Detection

Designated Object

• User specifies a template

• The system converts that template into an appropriaterepresentation to be tracked

Page 6: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Object Detection

Fixed Cameras

• Model the background using a reference image or a reference distribution

• Detect objects as changes with respect to the reference

Page 7: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Object Detection

Moving Cameras

• Align consecutive frames using thenow well-known techniques studiedin this class

• Use frame differencing betweenaligned frames to detect changesdesignated as new objects

Page 8: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Simple Tracker : Blob tracker

• Change-based tracker: – Approach

• Align video images• Detect regions of change• Track change blobs

– Problem with this approach is that it uses no appearanceinformation

• difficult to deal with stalled or close-by objects

Page 9: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Moving Blobs

Page 10: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Simple Tracker - Correlation Based

• Correlation-based tracker: – Approach

• Initialize the templates and the supports of foreground objects

• Estimate motion by correlation

– The problem with this approach is that it does not simultaneously compute the segmentation and appearance

• No accurate segmentation or region of support ⇒ may drift over time.

• Get confused by cluttered backgrounds

Page 11: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Problems with the simple trackers

• They lack the two key ingredients for optimal tracking:– State prediction– Optimal state updation

• Since measurements are never perfect --- each has some uncertainty associated with it --- optimal state prediction and updation need to take the uncertainties into account

• Furthermore, the object representation needs to be richer– Not just a change blob, or fixed template– Optimal method for updating the state

Page 12: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Kalman Filtering

• Assume that results of experiment(i.e., optical flow) are noisymeasurements of system state

• Model of how system evolves• Prediction / correction framework• Optimal combination

of system model and observations

Rudolf Emil Rudolf Emil KalmanKalman

Acknowledgment: much of the following material is based on theAcknowledgment: much of the following material is based on theSIGGRAPH 2001 course by Greg Welch and Gary Bishop (UNC)SIGGRAPH 2001 course by Greg Welch and Gary Bishop (UNC)

Page 13: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Simple Example

• A point whose position remains constant : x– Say a temperature reading

• Noisy measurement of that single point z1

• Variance σ12 (uncertainty σ1)

• Best estimate of true position

• Uncertainty in best estimate

11̂ zx = 11̂ zx =

221 1

ˆ σσ = 221 1

ˆ σσ =

Page 14: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Simple Example

• Second measurement z2, variance σ22

• Best estimate of true position

• Uncertainty in best estimate

( )121

11

21

11

2

ˆˆ

ˆ

22

21

21

22

21

22

21

xzx

zzx

−+=

+

+=

+σσσ

σσ

σσ

( )121

11

21

11

2

ˆˆ

ˆ

22

21

21

22

21

22

21

xzx

zzx

−+=

+

+=

+σσσ

σσ

σσ

22

21

1ˆ1

22

1ˆσσ

σ+

=22

21

1ˆ1

22

1ˆσσ

σ+

=

Page 15: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Online Weighted Average

• Combine successive measurements into constantly-improving estimate

• Uncertainty decreases over time

• Only need to keep current measurement,last estimate of state and uncertainty

We have essentially computed the Least Squares OR Minimum Variance OR Maximum Likelihood estimate of X given a number

of noisy measurements Z through an incremental method

Page 16: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Terminology

• In this example, position is state – in general, any vector

• State evolves over time according to a dynamic model or process model– (in this example, “nothing changes”)

• Measurements are related to the state according to a measurement model– (possibly incomplete, possibly noisy)

• Best estimate of state with covariance Px̂x̂

Page 17: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Tracking Framework• Very general model:

– We assume there are moving objects, which have an underlying state X

– There are measurements Z, some of which are functions of this state– There is a clock

• at each tick, the state changes• at each tick, we get a new observation

• Examples– object is ball, state is 3D position+velocity, measurements are stereo

pairs– object is person, state is body configuration, measurements are

frames, clock is in camera (30 fps)

Page 18: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Bayesian Graphical Model

1-kX

1-Zk

kX 1kX +

kZ 1k+ZK KMeasuement Model

State Variables: Those that tell us about objects & their states

But they are hidden, cannot be directly observed

Dynamic Model

Measurements: Can be directly observed

Are noisy, uncertain

Page 19: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Bayesian Formulation

1-k1-k1-k1-kkkkkk dx)z|x(p)x|x(p)x|z(p)z|x(p ∫κ=

)z|x(p kk Posterior probability after latest measurement

)x|z(p kk Likelihood of the current measurement

)x|x(p 1-kk Temporal prior from the dynamic model

)z|x(p 1-k1-k Posterior probability after previous measurement

κ Normalizing constant

Page 20: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

The Kalman Filter

• Key ideas: – Linear models interact uniquely well with Gaussian noise

• make the prior Gaussian, • everything else Gaussian and the calculations are easy

– Gaussians are really easy to represent• once you know the mean and covariance, you’re done

Page 21: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Linear Models

• For “standard” Kalman filtering, everything must be linear

• System / Dynamical model:

• The matrix Φk is state transition matrix

• The vector ξk represents additive noise, assumed to have covariance Q : N(0;Q)

111 −−− +Φ= kkkk xx ξ 111 −−− +Φ= kkkk xx ξ

)Q;x(N~x k1-kkk Φ )Q;x(N~x k1-kkk Φ

Page 22: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Linear Models

• Measurement model / Likelihood model:

• Matrix H is measurement matrix

• The vector μ is measurement noise, assumed to have covariance R :

)R;xH(N~zxHz

kkkk

kkkk μ+=)R;xH(N~z

xHz

kkkk

kkkk μ+=

);0(N μ

Page 23: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Position-Velocity Model

• Points moving with constant velocity• We wish to estimate their PV state at every time instant

[ ]0110

1

=

⎥⎦

⎤⎢⎣

⎡ Δ=Φ

⎥⎦

⎤⎢⎣

⎡=

H

t

x

kk

dtdxkx

[ ]0110

1

=

⎥⎦

⎤⎢⎣

⎡ Δ=Φ

⎥⎦

⎤⎢⎣

⎡=

H

t

x

kk

dtdxkx Position-Velocity State

Constant VelocityDynamic Model Matrix

Only position is directlyobservable

Page 24: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Prediction/Correction

• Predict new state

• Correct to take new measurements into account

1T

111

11 ˆ

−−−−

−−

+ΦΦ=′

Φ=′

kkkkk

kkk

QPP

xx

1T

111

11 ˆ

−−−−

−−

+ΦΦ=′

Φ=′

kkkkk

kkk

QPP

xx

( )( ) kkkk

kkkkkk

PHKIPxHzKxx

′−=

′−+′=ˆ ( )( ) kkkk

kkkkkk

PHKIPxHzKxx

′−=

′−+′=ˆ

Page 25: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Kalman Gain

• Weighting of process model vs. measurements

• Compare to what we saw earlier:

( ) 1TT −+′′= kkkkkkk RHPHHPK ( ) 1TT −+′′= kkkkkkk RHPHHPK

22

21

21

σσσ+ 2

221

21

σσσ+

Page 26: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Optimal Linear Filter

kkkkk zK)-(x̂K)(x̂ +′=+

Predicted state

MeasurementOptimalLinear Estimate

Under Gaussian assumptions, linear estimate is the optimal

Estimation Error: )(x~x)(x̂ kkk ++=+

kkkkkkkkk vK)-(x~KxI]-HKK[)(x~ +′++′=+

For an unbiased estimate: 0)](x~[E k =+ kkk HK-IK =′∴

)]-(x̂H-z[K)-(x̂)(x̂ kkkkkk +=+∴

kK Is obtained by minimizing the variance of the state estimate

Page 27: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Results: Position-Only Model

MovingMoving StillStill

[Welch & Bishop][Welch & Bishop]

Page 28: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Results: Position-Velocity Model

[Welch & Bishop][Welch & Bishop]

MovingMoving StillStill

Page 29: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Extension: Multiple Models

• Simultaneously run many KFs with different system models

• Estimate probability each KF is correct

• Final estimate: weighted average

Page 30: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Results: Multiple Models

[Welch & Bishop][Welch & Bishop]

Page 31: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Results: Multiple Models

[Welch & Bishop][Welch & Bishop]

Page 32: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Results: Multiple Models

[Welch & Bishop][Welch & Bishop]

Page 33: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Extension: SCAAT

• H be different at different times– Different sensors, types of measurements– Sometimes measure only part of state

• Single Constraint At A Time (SCAAT)– Incorporate results from one sensor at once– Alternative: wait until you have measurements

from enough sensors to know complete state (MCAAT)

– MCAAT equations often more complex, but sometimes necessary for initialization

Page 34: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

UNC HiBall

• 6 cameras, looking at LEDs on ceiling• LEDs flash over time

[Welch & Bishop][Welch & Bishop]

Page 35: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Extension: Nonlinearity (EKF)

• HiBall state model has nonlinear degrees of freedom (rotations)

• Extended Kalman Filter allows nonlinearities by:– Using general functions instead of matrices– Linearizing functions to project forward– Like 1st order Taylor series expansion– Only have to evaluate Jacobians (partial derivatives), not

invert process/measurement functions

Page 36: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Other Extensions

• On-line noise estimation

• Using known system input (e.g. actuators)

• Using information from both past and future

• Non-Gaussian noise and particle filtering

Page 37: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Data Association

• Nearest Neighbors– choose the measurement with highest probability given

predicted state– popular, but can lead to catastrophe

• Probabilistic Data Association– combine measurements, weighting by probability given

predicted state– gate using predicted state

Page 38: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Video based Tracking : Complexities

• In addition to position and velocity, object state may include:– Appearance, shape, specific object models : people, vehicles, etc.

• Camera may move in addition to the object– Track background as well as the foreground

• Measurement model and the associated likelihood computation is more complex:– Compute the likelihood of the presence of a head-n-shoulders person

model at a given location in the image

• Multiple objects need to be tracked simultaneously– Measurements need to be optimally associated with a set of models

rather than a single model as in the previous examples

Page 39: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Application - Tracking vehicles in aerial videos

ResultsVideo Stream

TrackingSystem

• The goals of a tracking system are to– detect new moving objects– maintain identity of objects, handle multiple objects and

interactions between them. e.g. passing, stopped, etc.– provide information regarding the objects, e.g. shape,

appearance and motion.

Page 40: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Tracking as a continuous motion segmentation problem

• Tracking problem ⇔ continuous motion segmentation problem: estimation of a complete representation of foreground and background objects over time.

• Complete representation (Layer) includes:– motion of objects and background– shape of objects and support– appearance of objects

• Key: constraints

Page 41: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Layer based motion analysis method

• Simultaneously achieve motion and segmentation estimation (EM algorithm)

– Estimate segmentation based on motion consistency

– Estimate motion based on segmentation

Page 42: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Motion layer representations -models/constraints

Local constraints Global constraints Multi-frame consistency

Motion

Smooth dense flow:Weiss 97

2D affine:Darrell91, Wang93, Hsu94, Sawhney96,Weiss 96, Vasconcelos97

3D planar: Torr99

2D rotation and translation &constant velocity:This paper

Segmentation

MRF segmentationprior:Weiss96, Vasconcelos97

Background+Gaussiansegmentation prior:

This paper - Section 2.1

Constant segmentation prior:

This paper - Elliptical shape prior

AppearanceConstant appearance:This paper

Page 43: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Dynamic Layer Representation

• Spatial and temporal constraints on the layer segmentation, motion, and appearance

• EM algorithm for maximum a posteriori estimation

• Layer ownership is constrained by a parametric shape distribution, instead of a local smoothness constraint. It prevents the layer evolving into arbitrary shapes, and enables tractable estimation over time.

Page 44: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Representation and constraints -segmentation and appearance

• Segmentation prior model• background + elliptical shapes• constant value over time

• Appearance model -• constant value over time

},{ ttt sl=Φ

β

Layer j

γ

tA

Background layer

Page 45: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Representation and constraints -motion

• Motion model – motion

• foreground– translation + rotation– constant velocity model

• background– planar surface

),( ttt u ωΘ =

tu

Page 46: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

MAP estimation

)_,,,,|_,,(

1111 −−−− tttt

tttt

priorshapeappearancemotionimageimagepriorshapeappearancemotionP

motion,appearance,shape prior

motion,appearance,shape prior

motion,appearance,shape prior

imaget-1 imaget imaget+1

Page 47: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

MAP estimation - formulation

• Notation– current image is . Current state is

.

• Estimation

tI ]A,,[ tttt ΦΘΛ =

),|(),,|(argmax

),,|(argmax

1111

11

−−−−

−−

= ttttttt

tttt

PP

P

t

t

III

II

ΛΛΛΛ

ΛΛ

Λ

Λ

priorlikelihood

Page 48: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Optimization using EM algorithm

• The general Expectation Maximization algorithm– observation and parameter– objective function:

– equivalent to iteratively improving conditional expectation

• For the dynamic layer tracker:

• Optimize over

)(log],|)|,([log)|( θθθθθ PyyxPEQ +′=′

)()|(argmax θθθ

PyP

y θ

)|(log],,,|),,|,([log 11111 −−−−− +′= ttttttttttt PIIIzIPEQ ΛΛΛΛΛΛ

tΛQ

Page 49: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

• Optimization over motion, segmentation, and appearance correspond to the following three steps:

– layer motion estimation based on current segmentation and appearance ⇒ weighted correlation or direct method

– layer segmentation estimation ⇒ competition between motion layers

– layer appearance estimation ⇒ Kalman filtering of appearance

Optimization - 3 steps

Page 50: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Optimization - flow chart

estimate motion

tΘestimate shapeprior

update ownershipjih ,

estimate appearancetA

update ownershipjih ,

update ownershipjih ,

frame t-1

frame t

frame t+1

Page 51: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Optimization - illustration

appearance t-1

frame t

motion estimation

shape estimation

appearance estimationshape prior t-1

motion t-1

appearance t

shape prior t

motion t

Page 52: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Optimization - equations

• Motion estimation– weighted SSD

• Ownership estimation - gradient method

• Appearance estimation

)//1(

/)(/))(())(( 2

,2

2,

2,

,IjiA

IitjiAijjtijjt h

xIhxTAxTA

σσ

σσ

+

+=

2,1,

3,

2.,

1

0,

,

,,

,

/)(

/))(()()(

))()((

lsjtjt

jtyji

n

iijt

iijt

ijtiji

jt

ss

syxLxDxL

xLxDhsf

σ

γ

=

−−

−−

=∂∂

2,1,

3,

2.,

1

0,

,

,,

,

/)(

/))(()()(

))()((

lsjtjt

jtxji

n

iijt

iijt

ijtiji

jt

ll

lyxLxDxL

xLxDhlf

σ

γ

=

−−

−−

=∂∂

Page 53: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Inference of object status

• A state transition graph is designed to

– trigger events such as object initialization, object elimination

– infer object states such as moving, stopping, two objects that are close to each other, etc.

Page 54: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Inference of Object Status

ConditionsNS = normal SSD scoreOB = out of scopeLT = NM for a long timeZM= zero motion estimationNB = new blob, no object covering a blobNM = no motion blob covering the objectSI = significant increase of SSD

disappearnew moving

stop

occluded

NB

NM & SI

NM &!SI&ZM

!NM&NS

OB|LT

!NS|OB|LT

OB

!NM&NS

NM | OB

!NM

NM&!SI&!ZM

NM |{!NM&!NS}

NM&NS

!NM & !OB

Page 55: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Implementation - Sarnoff Layer Tracker

• Performance:– Originally developed on a PC, ported to SGI Octane. 20-25

Hz for one object over a single processor.

Airborne Video Surveillance System (tracking component)

SarnoffVFE 200 SGI Octane

Video Stream

Page 56: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Results

• Turning

Page 57: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Results

• Turning

(a) (b) (c)

Page 58: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Results

• Passing - opposite directions

Page 59: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Results

• Passing - opposite directions

(a) (b) (c)

Page 60: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Results

• Passing - the same direction

Page 61: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Results

• Passing - the same direction

(a) (b) (c)

Page 62: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Results

• Stop, Passing

Page 63: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Results

• Stop, Passing

(a) (b) (c)

Page 64: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Implementation - Sarnoff Layer Tracker

• Motion estimation:– 95% of computation is for motion estimation. Currently,

weighted SSD correlation is used. Searching in a 13x13 window at half resolution, for 3 different angles. The size of the object is around 40x40 pixels.

• Ownership estimation– change image is integrated into the formulation to further

improve the robustness.

• Appearance estimation– appearance model for the background is not computed,

instead, the previous image is used.

Page 65: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

An Alternative Appearance Model

• In the previous model, appearance gets incrementally averaged over time since it is part of the state vector

• A more sophisticated appearance model allows for averaging as well as keeping up with frame-to-frame appearance changes:

– Jepson et al.’s WSL model– A mixture model of appearance– Estimated incrementally using online EM

Page 66: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

WSL Adaptive Model in 1D

Mixture model for current data (4 dof):

)()|()|(),,|( 11 tllttwwttsstttt dpmddpmdpmddp ++= −− qmq

stable parameters),( 2

,, tstst σμ=q ),,( lwst mmm=mmixing probabilities

ts,μ

2,tsσ

1−td td

]11[ ,dt −∈

2wσ

Wandering Process: constant variance

Lost Process

Stable Process:variance

Page 67: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

On-Line Approximate EM

One EOne E--Step:Step: Compute data ownerships only at current time

),,|(),|(

)(111

111,,

−−−

−−−=tttt

tttjtjttj ddp

ddpmdo

mqq

},,{ lswj∈

One MOne M--Step:Step: Update weighted ith-order data moments

,)( )0(,, tjktj Mdm =

)0(,

)1(,

,ts

tsts M

M=μ 2

,)0(,

)2(,2

, tsts

tsts M

Mμσ −=

Updated mixing probabilities (0th order moments):

Updated mean and variance of stable process:

,)1()( )(1,,

)(,

itj

itttj

itj MddoM −−+= αα

},,{ lswj∈

},,{ lswj∈

Page 68: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Estimation of Motion Parameters

)|(log),,|(log)( 111 −−− += ttttttt pDADLO uuuulikelihood prior

where:warp parameters:data at time t-1:

appearance model:parametric motion:

1-tR∈− −

To estimate the motion model parameters we maximize the sum of log likelihood and log prior :

= xx )}1,({1 tdDt

),( tttA mq=

);( 1 ttt uxwx −=

tu

Page 69: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Optimization Details

Data Likelihood:

data from time t is warped back to t-1 and compared to predictions from the tracking region at time t-1.

∏∈

−−− −=1

))1,(,|)),;(((),,|( 111

-

tdAtdpDADL tttttt

tRx

xuxwu

Motion Prior:),(),()|( 2

11201 σσ −− −= ttttt GGp uuuuu

slow smooth

Fitting process for is similar to fitting mixture models for flow (Jepson & Black, 1993).

tu

Page 70: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Real-Time Tracking

Page 71: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

AdaptiveBackground

Modeling

ForegroundDetection

Frame-to-frame

Tracking

StateMachine

Filtered Representation

Filtered Background Model

Detected change objects

Major Components of a Tracker

Page 72: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

System state at time t At time (t+1) System state at time (t+1)

Object (green box) as seen at time t. (latest model of appearance)

Object appearance as learnt from recent past. (learnt model of appearance)

Probabilistic visibility mask, brighter the pixel, more likely that it belongs to the object

• Velocity estimate

• Depth if available

motion estimation

appearance estimation

visibilityestimation

• Velocity estimate

• Depth if available

Latest appearance model

Updated learnt model

Updated visibility mask

Tracker Block Diagram

Occlusion handling

Page 73: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

System state at time t=1 System state at time t=8 System state at time t=16 System state at time t=27

Occlusion is detected at this frame

Note learnt model is much more immune to occlusions than the latest model.

The appearance models and visibility mask are still frozen to t=8 because of occlusion

The object reappears after occlusion, and the models and visibility mask are updated

Sample Progress of the Tracker

Page 74: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Tracker Features

• Non-parametric distribution based background representation.– Resilient to environmental effects like wind-induced motion, heat-

induced scintillation etc.

• Foreground extraction based on pyramid filters and flow.– Tunable for different scenarios: outdoors, indoors.

• Comprehensive tracking based on appearance, motion and shape.– Automatically adapts to smooth and sudden changes of

appearance.– Automatically weights appearance and shape matching.– Precise motion estimation based on optical flow.

• State machine that exploits appearance, motion and shape.– Handles occlusions, and confusing events with multiple objects.

Page 75: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Example: Outdoors

Page 76: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Example: Indoor Overhead

Page 77: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Example: Airport Overhead

Page 78: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Example: Airport (Light Traffic)

Page 79: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Example: Airport Sequence

Page 80: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Example: Hallway Sequence

Page 81: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Example: Hallway Sequence

Page 82: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

3D Tracking with Presence of Clutter and Multi-Camera Handoff

Video of Camera 1 and Camera 2

Camera 2

Handing-off from camera 1 to camera 2

Camera 1

Page 83: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

3D Tracking in Outdoor Scenarios

Original videoVideo with entire mob being tracked simultaneously

• Each color represents a different person in the image

• Note the 3D tracker can distinguish between people and their shadows

Depth Map Video

Page 84: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

3D Tracking in Outdoor Scenarios

Original videoVideo with people and vehicles being tracked simultaneously

Each color represents a different person/vehicle in the image

Depth Map Video

Page 85: Moving Object Tracking · The Tracking Problem : Issues • Object Detection: Designated or automatically detected • Object State Instantiation: Object representation – Position,

Recommended