Vision-based 3D Bicycle Tracking using Deformable Part Model
and Interacting Multiple Model Filter
May 11, 2011
Hyunggi Cho1, Paul Rybski1,2, and Wende Zhang3
1Electrical and Computer Engineering
School of Engineering
Carnegie Mellon University
3The Electrical and Controls
Integration Lab.
General Motors R&D
2Robotics Institute
School of Computer Science
Carnegie Mellon University
Outline
Motivation and Overview
Bicycle Detection
Bicycle Tracking
Experimental Results
Conclusion and Future work
Motivation
Motivation
- In 2009, 630 bicyclists were killed and 51,000 were injured in traffic accidents in the United States*.
- Bicyclists and pedestrians are the most vulnerable traffic participants.
- There is less research on bicyclist detection and tracking compared to that of pedestrians.
*http://www.nhtsa.gov
Movie clip: Bicycle messengers in New York City (Youtube)
Sensors on Cars
Source : http://www.tartanracing.org
System Overview
Input : video
Track bicycles using a single video camera mounted on a vehicle
System
output : position & velocity
System block diagram
BicycleDetector
BicycleTracker
Bicycle’sposition & velocity
Bicycle Detection – Deformable Part Model HOG Detector Eight view-based bicycle detection
root filters
coarse resolution
part filters
finer resolution
deformation
models
P. Felzenszwalb, R. Girshick, D. McAllester, D. Ramanan, Object Detection with Discriminatively Trained Part Based Models, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010
DPM HOG Detector – Object hypothesis
Bicycle detection process
P. Felzenszwalb, R. Girshick, D. McAllester, D. Ramanan, Object Detection with Discriminatively Trained Part Based Models, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010
DPM HOG Detector – Performance Analysis Examples of bicycle detection
Test images from Google image
DPM HOG Detector – Performance Analysis
Terminology
Eight view
Precision-Recall Curve (VOC2009 + Ours)
Recall (True Positive Rate)
Precision
)( FNTP
TP
P
TPrecall
)( FPTP
TPprecision
True Positive
False Negative
False Positive
Total No. of Positive
Average Precision( Area Under Curve )
TP
FNFPP
AP
Training Set : 350 positive / 3300 negative Test Set : : VOC2009 ‘val’ dataset
Overview of our single bicycle tracking system
Prediction stage Update stage
y
x
Kalman filter-based tracking
][ yxyxx state space :
2D image space
Overview of our single bicycle tracking system
Model
Dynamic system model
v
y
x
y
x
y
x
y
x
~
10
01
00
00
0000
0000
1000
0100
kkk
kkk
vxhz
wFxx
)(1
),0(~
),0(~
),0(~ 00
kk
kk
RNv
QNw
Nx
Motion model : constant velocity
kkk wFxx 1
y
x
: Dynamic equation
: Measurement equation
: Initial state
: Process noise
: Measurement noise
Overview of our single bicycle tracking system
Measurement model : perspective projection
rotation matrix
translation vector
focal length
optical center
Rtf
cc vu ,
kkk vxhz )(
343331
1413111 ayaxa
ayaxa
w
uh
kk
kk
343331
2423212 ayaxa
ayaxa
w
vh
kk
kk
1
0
34333231
24232221
14131211
k
k
y
x
aaaa
aaaa
aaaa
1
0|
100
0
0
k
k
c
c
y
x
tRvf
uf
w
v
u
Horizon
Image plane
height
Image
IMM - Choosing a model set
Constant Velocity Coordinated Turn
Constant Velocity Simplified Bicycle with
CV and CY angle
Model Set I Model Set II Model Set IIIGM
GM
CV
CA
Constant Velocity Constant Acceleration
CV
CT
CV
SB
IMM - Choosing a model set
Constant Velocity model :
Simplified Bicycle model :
v
y
x
y
x
y
x
y
x
~
10
01
00
00
0000
0000
1000
0100
][ yxyxx
IMM - Performance analysis
We tested the IMM method on the GM bicycle dataset
Test Set : 6 sequences with a stationary GM test vehicle
Data statistics : Size : 320x240 , FPS : 10~12 , No. of bicycles : 1
IMM Tracking performance
Details of six bicycle sequences ( SM vs. IMM )
Seq. ego-vehicle bicycle RMSE(SM) RMSE(IMM)
‘seq1’ stationary laterally 0.0183 0.0216
‘seq2’ stationary longitudinally 6.6207 6.6196
‘seq3’ stationary randomly 0.1515 0.1443
‘seq4’ moving laterally 2.3493 2.3860
‘seq5’ moving longitudinally 7.0884 6.860
‘seq6’ moving randomly 11.0929 10.6281
IMM - Performance analysis
Sequence 3 case
Multiple bicycle tracking using a Rao-Blackwellized particle filter
Single bicycle tracking : We solved this problem.
Data association : Given a measurement, which target produced it, if any ?
Unknown number of targets : How many bicycles are there ?
Multiple bicycle tracking problem
In our multiple bicycle tracking case
)|(),|()|,( :1:0:0:1:0:1:0:0 tttttttt yspsyrpysrp Particle filterKalman filter
},{ ttt ces
TTttt rrr ][ ,1, Joint state vector
: Data association indicator
:Target visibility indicator
tc
te
Simo Särkkä, Aki Vehtari, and Jouko Lampinen (2007). Rao-Blackwellized Particle Filter for Multiple Target Tracking. Information Fusion Journal, Volume 8, Issue 1, Pages 2-15
Multiple bicycle tracking using a Rao-Blackwellized particle filter
Only one target can die
is associated with : (a) Clutter(b) One of the existing targets(c) A newborn target
ky
All possible events between two measurements and 1ky ky
Example
t-2
t-1
t
t-2
t-1
t
y1
y2
y3
: Target
: Measurement
: Trajectory
Particle filter for data association problem
Experimental Results
We tested our detection/tracking system on our bicycle dataset
Test Set : A challenging sequence from a moving Boss (so called ‘Free for all’)
Data statistics : Size : 320 x 240 , Frame rate : 13~15 frame per second
Sensor coverage area
Tracking performance
15 m
5 m
0 m
4 m
Minimum pixel sizeHOG Detector : 32x64
4 m
Experimental Results - data collection
Rank & Rate Description Illustration
10(3.9%)
Motorist Overtaking-OtherThe motorist was overtaking a
bicyclists.
9(4.3%)
Bicyclist Left Turn in front of trafficThe bicyclist made a left turn in front
of traffic travelling in the same direction.
8(4.4%)
Ride Out At MidblockThe bicyclist entered the roadway at a
shoulder or curb midblock location.
US Bicyclists Crash Types – Top 10covering 61% database samples
W.H. Hunter, W.E. Pein, and J.C. Stutts, Bicycle Crash Types: A 1990's Informational Guide, Publication No. FHWA-RD-96-104, Federal Highway Administration, Washington, DC, April, 1997
Experimental Results - data collection
Rank & Rate Description Illustration
7(4.7%)
Motorist Right TurnThe motorist was making a right turn and the bicyclist was riding in either
the same or opposing direction.
6(5.1%)
Ride Out At Residential DrivewayThe bicyclist entered the roadway from
a residential driveway or alley.
5(5.9%)
Motorist Left Turn– Facing BicyclistThe motorist made a left turn while
facing the approaching bicyclist.
4(6.9%)
Ride Out At MidblockThe motorist was entering the roadway
from a driveway or alley
W.H. Hunter, W.E. Pein, and J.C. Stutts, Bicycle Crash Types: A 1990's Informational Guide, Publication No. FHWA-RD-96-104, Federal Highway Administration, Washington, DC, April, 1997
Experimental Results - data collection
Rank & Rate Description Illustration
3(7.1%)
Ride Out At Intersection - OtherThe crash occurred at an intersection, signalized or uncontrolled, at which the
bicyclist failed to yield.
2(9.3%)
Drive Out At Stop SignThe crash occurred at an intersection
at which the motorist was facing a stop sign.
1(9.7%)
Ride Out At Stop SignThe crash occurred at an intersection
at which the bicyclist was facing a stop sign or flashing red light.
We categorized the upper scenarios into 4 different classes in terms of bicycle motion patterns !!!
W.H. Hunter, W.E. Pein, and J.C. Stutts, Bicycle Crash Types: A 1990's Informational Guide, Publication No. FHWA-RD-96-104, Federal Highway Administration, Washington, DC, April, 1997
Experimental Results - data collection
Experimental Results – Performance analysis
Scenario – Random moving case (‘Free for all’)
2D Bounding box {view (x coordinate, y coordinate)} Uncertainty level
Experimental Results – Performance analysis
Scenario – Random moving case with 3D visualization
2D Bounding box {view (x coordinate, y coordinate)} Uncertainty level
Summary and Future Work
Summary Data collection
- Based on bicycle accident statistics
Detection part- Applied DPM HOG detector into a multiple bicycle tracking system
Tracking part- Incorporate Interacting Multiple Model (IMM) algorithm into our multiple bicycle tracking system to exploit several types of motion models- RBPF data association algorithm
Future work Real-time C++ implementation ( > 10fps)
Integration the system into the perception system of our autonomous vehicles at CMU
Q&A
Single bicycle tracking using an IMM
True motion of a bicycle cannot be exactly modeled by just one model, only be sufficiently approximated by using several motion models for representing dynamic driving behaviors of a target (i.e., maneuverings of a bicycle).
The IMM filter runs several motion models in parallel and estimates a state by computinga weighted sum of several filter results which are based on different motion models.
Main idea of Interacting Multiple Model (IMM)
Integral HOG Detector - Performance Analysis II
Related works
Vision-based Bicycle Detector
Publication Sensors Features Attention Focusing Stage Classification stage
Gavrila IV2004 Stereo Edge map
Stereo-based depthChamfer matching
Texture classification
Papageorgiou
IJCV2000Monocular Haar wavelet
Can add motion/stereo modules for preprocessing
SVM classifier on Haar wavelet features
Viola & Jones
CVPR2001Monocular Haar-like wavelet NA AdaBoost
Dalal & Trigg
CVPR2005Monocular HOG NA Linear SVM on HOG
Zhu & Avidan
CVPR2006Monocular Integral HOG NA
AdaBoost with linear SVM as a weak
classifier
Miko.
ECCV2004Monocular
SIFT-like orientation feature
NA AdaBoost
Wu & Nevatia
ICCV2005Monocular Edgelets NA
AdaBoost with hard-coded mid-level features
Felzenszwalb
CVPR2008Monocular HOG NA
Deformable part model with LSVM