K O U R O S HM E S H G IPROGRESS REPORT TOPIC
Occlusion Aware Particle Filter Tracker to Handle Complex and
Persistent Occlusions usingMultiple Feature Fusion
To: Ishii Lab Members,Dr. Shin-ichi Maeda, Dr. Shigeuki Oba,
And Prof. Shin Ishii
9 MAY 2014
TRACKING APPLICATIONS
K O U R O S H M E S H G I – I S H I I L A B - D E C 2 0 1 3 - S L I D E 2
MAIN APPLICATIONS
Surveillance Public Entertainment
Robotics Video Indexing
Action Recog.
TRACKING CHALLENGES
K O U R O S H M E S H G I – I S H I I L A B - D E C 2 0 1 3 - S L I D E 3
MAIN CHALLENGES
Varying ScaleClutterDeformation
OcclusionIlluminationAbrupt Motion
Goal: Define p(Xt|Y1,…,Yt) given p(X1)
BAYESIAN TRACKING
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 4
X1 X2 … Xt
Y1 Y2 … Yt
States: Target Location and Scale
Observations: Sensory Information
PARTICLE FILTER TR.
INTRODUCTION• • • • • • • • • • • • • • • • • • • • • • • •
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 6
INPUT IMAGEFrame: t
RGB Domain
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 7
INPUT DEPTH MAPFrame: t
Depth Domain
Close Far
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 8
SENSORY INFORMATIONFrame: t
Sensory Information
, ,{ , }t rgb t d tI I I
Observation
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 9
STATE REPRESENTATION & OBSERVATION MODEL
Frame: t
State
{ , , , }t t t t tB x y w h{ }t tX B
( ; )t t tY g I B
w
h
(x,y)
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 1 0
FEATURES
Feature Set
1{ ,..., }MF f f
Color
Shape Edge
Texture
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 1 1
TEMPLATEFrame: 1
Template
1 1,1 ,1{ ,..., }M
f1 fj fM
1 ,1 1{ }Mi i
1 1{ ( )}if Y
… …
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 1 2
PARTICLES INITIALIZATION
Frame: 1
Particles
, ,{ }k t k tX B1,2, ,k N
Initialized Overlapped
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 1 3
MOTION MODELFrame: t
Motion Model
, , ,k t k t k tB B
→ t + 1
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 1 4
FEATURE EXTRACTIONFrame: t + 1
Feature Vectors
, 1( )i k tf Y
f1 f2 fM
X1,t+1
X2,t+1
XN,t+1
…
…
K O U R O S H M E S H G I – I S H I I L A B - M A R 2 0 1 4 - S L I D E 1 5
FEATURE FUSIONFrame: t
Probability of Observation( | , )t t tp Y X ,1
( ( ) | , )M
i i t t i tip f Y B
( ( ) | , )t t tp f Y B ,1( ),
M
i i i t i tip D f Y
,1exp ( ),
M
i i i t i tiD f Y
,1
exp ( ),M
i i i t i tiD f Y
Each Feature(.)if(.)iD
i Indep
enden
ce
Assum
ption
!
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 1 6
PROB. CALCULATIONFrame: t + 1
Particles
Brighter = More Probable
,k tp
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 1 7
TARGET ESTIMATIONFrame: t + 1
Feature Vectors
1 1ˆ | ,...,t t tB B Y Y E
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 1 8
MODEL UPDATEFrame: t + 1
New Model
Model Update
1ˆ ˆ( ; )t t tY g I B
1ˆ ˆ( )t i tf Y
, 1 , 1
,
ˆ
(1 )i t i i t
i i t
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 1 9
RESAMPLINGFrame: t + 1
Proportional to Probability
1( | )t tp X X1
2
345
6
7
PARTICLE FILTER TR.
CHALLENGES• • • • • • • • • • • • • • • • • • • • • • • •
PFT ISSUES
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 2 1
Appearance Changes
Model Drift
Deficient Feature Space
Uninformed Search
Optimized Feature Selection
Approximation of Target
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 2 2
APPEARANCE CHANGES
Same Color Objects
Background Clutter
Illumination Change
Shadows, Shades
Use Depth!
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 2 3
MODEL DRIFT PROBLEM
Templates Corrupted! t
Handle Occlusion!(No Model Update During Them)
DEFICIENT FEATURE SPACE
* Local Optima of Feature Space
* Feature Noise
* Feature Failures
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 2 4
Regularization
Non-zero Values
Normalization
PERSISTENT OCCLUSION
Particles Converge to Local Optima / Remains The Same Region
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 2 5
Advanced Motion Models
(not always feasible)
Restart Tracking
(slow occlusion recovery)
Expand Search Area!
DYNAMICS…
* The Search is not Directed
* Neither of the Channels have Useful Information
* Particles Should Scatter Away from Last Known Position
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 2 6
Occlusion!
OCCLUSIONdo not address occlusion explicitly
maintain a large set of hypotheses
computationally expensive
direct occlusion detection
robust against partial & temp occ.
persistent occ. hinder tracking
GENERATIVE MODELSDISCRIMINATIVE
MODELS
Dynamic Occlusion: Pixels of other object close to camera
Scene Occlusion: Still objects are closer to camera than the target object
Apparent Occlusion: Due to shape change, silhouette motion, shadows, self-occ
UPDATE MODEL FOR TARGET TYPE OF OCCLUSION IS IMPORTANT KEEP MEMORY VS. KEEP FOCUS ON THE TARGET
Combin
e
Them!
PTO partial occlusion SAO self- or articulation occlusion TFO temporal full occlusion - shorter than 3
frames PFO persistent full occlusion CPO complex partial occlusion - including “split
and merge” and permanent changes in a key attribute of a part of target
CFO complex full occlusion
OCCLUSION TYPES
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 2 8
[Zhao & Nevatia, 04] Occlusion Indicator: Ratio of FG/BKG
[Wu & Nevatia, 07] Handle Occlusion using Appearance Model
[de Villiers et al., 12] Switch Tracker in the case of Occlusion
[Song & Xiao, 13] Occlusion Indicator: New Peak in HOD or Reduction of the Size of Main Peak
LITERATURE REVIW
Many other papers handle occlusions as the by-product of their novel trackers
OCCLUSION AWARE PFT
SOLUTION• • • • • • • • • • • • • • • • • • • • • • • •
Motion Model
Resampling
Target Estimation
Calculate Likelihood
PR
OP
OS
ED
MO
DIF
ICA
TIO
N
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 3 1
Initialization
Model Update
Observation
Occlusion Flag?
Constant Likelihood
Occlusion Estimation
Occlusion Threshold>?
YES
YES
NO
NO
Occlusion Flag (for each particle)
Observation Model
No-Occlusion Particles Same as Before
Occlusion-Flagged Particles Uniform Distribution
OCCLUSION AWAREPARTICLE FILTER FRAMEWORK
( | ) ( | , , )t t t t t tp Y X p Y B Z ( | ) (1 ) ( | , 0, ) ( | , 1, )t t t t t t t t t t t tp Y X Z p Y B Z Z p Y B Z
,k tZ
( | , 1, ) 1t t t tp Y B Z
,1( | , 0, ) exp ( ),
M
t t t t i i i t i tip Y B Z D f Y
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 3 2
Position Estimation of the Target
Occlusion State for the Next Box
TARGET ESTIMATION
1 1
, , , ,1
ˆ [ | ,..., ]
( | , , )
t t t occ
N
j t j t j t j t t occj
Z u Z Y Y
u Z p Y B Z
E
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 3 3
1
, , , ,
ˆ [ | ,..., , 0]
( | , 0, )
t t t t
j t j t j t j t tj
B B Y Y Z
B p Y B Z
E
J '
1
0
1
0 a
( )u x
( )u x a0a
x
x
Model Update (Separately for each Feature)
Modified Dynamics Model of Particle
UPDATE RULE
11
1 1
ˆ( ) ,( )
ˆ ˆ( ) (1 ) ( ) ,
t t occt
t t t occ
f Zf
f Y f Z
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 3 4
1 1 1 1 1( | ) ( , | , ) ( | ) ( | )t t t t t t t t t tp X X p B Z B Z p B B p Z Z
OA-PF DYNAMICS
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 3 5
Occlusion!
OA-PF DYNAMICS
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 3 6
Occlusion!
GOTCHA!
OA-PF DYNAMICS
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 3 7
Quick Occlusion Recovery
Low CPE
No Template Corruption
No Attraction to other Object/ Background
CO
LO
R
(HO
C)
TE
XT
UR
E
(LB
P)
ED
GE
(L
OG
)
2D P
RO
J. (B
ET
A)
3D S
HA
PE
(P
CL
Σ)
FEATURES
DE
PT
H
(HO
D)
GR
AD
IEN
T
(HO
G)
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 3 8
& DISCUSSION
RESULTS• • • • • • • • • • • • • • • • • • • • • • • •
Princeton Tracking Dataset
DATASET( )
5 Validation Video with Ground Truth95 Evaluation Video
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 4 0
EXPERIMENT
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 4 1
OAPFT (Proposed, with different feature sets)
OI+SVM (SVM tracker with Occlusion Indicator)
• State-of-the-art RGBD tracker
ACPF (Adaptive Color Particle Filter)
• Traditional Particle Filter tracker
STRUCK (Structured Output SVM Tracker)
• State-of-the-art RGB tracker, Successful for Occlusion Handling
PASCAL VOC: Overall Performance
CRITERIA I
1
1
*1
* *1 1 1
*1 1
*1 1
ˆ
ˆ ˆ, 0
ˆ1 , 1
ˆ1 ,
t
t
t
t t t
t t t
t t
B B
B B Z Z
S Z Z
Z Z
0 1ott oS t AUC
toS
ucc
ess
Overlap Threshold
0
1
1
Area Under Curve
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 4 2
RESULTS
K O U R O S H M E S H G I – I S H I I L A B - M A R 2 0 1 4 - S L I D E 4 3
1
1
Success Plot
Overlap Threshold
Succ
ess
Rat
e
1
1
Mean Central Point Error: Localization Success
Mean Scale Adaption Error
CRITERIA II
* 2 * 2
1ˆˆ( ) ( )
T
t t t ttw w h h
SAET
* 2 * 2
1ˆ ˆ( ) ( )
T
t t t ttx x y y
CPET
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 4 4
ˆˆ ˆ ˆ ˆ{ , , , }t t t t tB x y w h * * * * *{ , , , }t t t t t
B x y w h
Estimated Ground Truth
RESULTSCenter Positioning Error
400
50Frames
CP
E (
pixe
ls)
RESULTSScale Adaptation Error
140
50Frames
SAE
(pi
xels
)
FP happens when a tracker doesn’t realize that the target is occluded.
MI happens when the target is visible but the tracker fails to track it as if the target is still in an occlusion state
MT the estimated bounding box has nothing in common with ground truth box
FPS execution time in frames per second
CRITERIA III
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 4 7
RESULTS
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 4 8
Tracker
AUC
CPE
SAE
MI FP MTFPS
BCDEGST (proposed)
76.50
9.59
7.32
0.0 2.4 0.0 0.9
ACPF (Nummiaro ‘03)
27.55
90.38
35.27
12.6
0.031.0
1.4
STRUCK (Hare ‘11)
46.67
68.74
26.61
12.6
0.064.4
13.4
OI+SVM (Song ‘13)
69.15
9.68
12.04
0.420.0
0.8 0.4
FUTURE WORKS
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 4 9
More Resilient Features + Scale
Adaptation
Active Occlusion Handling
Measure the Confidence of
each Data Channel
Adaptive Model Update
QUESTIONS?Thank you for your time…