Date post: | 18-Dec-2015 |
Category: |
Documents |
View: | 218 times |
Download: | 3 times |
Optic Flow and Motion Detection
Cmput 498/613Cmput 498/613
Martin JagersandMartin Jagersand
Readings: Papers and Readings: Papers and chapters on 613 web page chapters on 613 web page
Image motion
• Somehow quantify the frame-to-frame Somehow quantify the frame-to-frame differences in image sequences.differences in image sequences.
1.1. Image intensity difference.Image intensity difference.
2.2. Optic flowOptic flow
3.3. 3-6 dim image motion computation3-6 dim image motion computation
Motion is used to:
•Attention:Attention: Detect and Detect and direct using eye and direct using eye and head motions head motions
•Control:Control: Locomotion, Locomotion, manipulation, toolsmanipulation, tools
•Vision:Vision: Segment, depth, Segment, depth, trajectorytrajectory
MOVING CAMERAS ARE LIKE STEREO
Locations ofpoints on the object(the “structure”)
The change in spatial locationbetween the two cameras (the “motion”)
Classes of motion
•Still camera, single moving objectStill camera, single moving object
•Still camera, several moving objectsStill camera, several moving objects
•Moving camera, still backgroundMoving camera, still background
•Moving camera, moving objectsMoving camera, moving objects
The optic flow field
•Vector field over the image:Vector field over the image:
[u,v] = f(x,y), u,v[u,v] = f(x,y), u,v = Vel vector, = Vel vector, x,y = x,y = Im posIm pos
•FOE, FOCFOE, FOC Focus of Expansion, Contraction Focus of Expansion, Contraction
Optic/image flow
•Assumption: Image intensities from object points Assumption: Image intensities from object points remain constant over time.remain constant over time.
Im(x + î x; y + î y; t + î t) = Im(x; y; t)
Taylor expansion of intensity variation
Keep linear termsKeep linear terms• Use constancy assumption and rewrite:Use constancy assumption and rewrite:
• Notice: Linear constraint, but no unique solutionNotice: Linear constraint, but no unique solution
à@t
@Im = @x@Im; @y
@Imð ñ
á î xî y
ò ó
0 = @x@Imî x + @y
@Imî y + @t@Imî t
Im(x + î x;y + î y; t + î t) = Im(x;y; t) + @x@Imî x + @y
@Imî y + @t@Imî t + h:o:t:
Im(x + î x;y + î y; t + î t) = Im(x;y; t) + @x@Imî x + @y
@Imî y + @t@Imî t + h:o:t:
Aperture problem
• Rewrite as dot productRewrite as dot product
• Each pixel gives one equation in two unknowns:Each pixel gives one equation in two unknowns:f f . . n n = k= k
• Notice: Can only detect vectors normal to gradient Notice: Can only detect vectors normal to gradient directiondirection
• The motion of a line cannot be recovered using only The motion of a line cannot be recovered using only local informationlocal information
à @t@Im
= @x@Im ;@y
@Imð ñ á îxîy
ò ó= r Imá îx
îy
ò ó
fn
f
ît
The flow continuity constraint
• Flows of nearby patches are Flows of nearby patches are nearly equalnearly equal
• Two equations, two Two equations, two unknowns:unknowns:f f . . nn11 = k= k11
f f . . nn22 = k= k22
• Unique solution exists, Unique solution exists, provided provided nn11 and and nn22 not not
parallelparallel
fn
f
f
n
f
Sensitivity to error
• nn11 and and nn22 might be might be almostalmost
parallelparallel• Tiny errors in estimates of Tiny errors in estimates of kk’s ’s
or or nn’s can lead to huge errors ’s can lead to huge errors in the estimate of in the estimate of ff
fn
f
fn
f
Using several points
•Typically solve for motion in 2x2, 4x4 or larger Typically solve for motion in 2x2, 4x4 or larger patches. patches.
•Over determined equation system:Over determined equation system:
dI = MudI = Mu•Can be solved in e.g. least squares sense using Can be solved in e.g. least squares sense using matlab matlab u = M\dIu = M\dI
...à
@t@Im
...
0
@
1
A =
......
@x@Im
@y@Im
......
0
@
1
A î xî y
ò ó
Flow vectors
•Norbert’s trick: Use an mpeg-card to speed Norbert’s trick: Use an mpeg-card to speed up motion computationup motion computation
Other applications:
•Recursive depth recovery: Kostas Danilidis and Recursive depth recovery: Kostas Danilidis and Jane MulliganJane Mulligan
•Motion control (we will cover)Motion control (we will cover)
•SegmentationSegmentation
•TrackingTracking
Lab:
• Purpose:Purpose:– Hands on optic flow experienceHands on optic flow experience– Simple translation trackingSimple translation tracking
• Posted on www page.Posted on www page.• Prepare through readings: Prepare through readings:
(on-line papers.(on-line papers.• Lab tutorial: How to capture.Lab tutorial: How to capture.• Questions: Email or course Questions: Email or course
newsgroupnewsgroup• Due 1 week.Due 1 week.
Organizing different kinds of motion
Two ideas:Two ideas:
1.1. Encode different motion structure in the M-Encode different motion structure in the M-matrix. Examples in 4,6,8 DOF image plane matrix. Examples in 4,6,8 DOF image plane tracking. 3D a bit harder.tracking. 3D a bit harder.
2.2. Attempt to find a low dimensional subspace for Attempt to find a low dimensional subspace for the set of motion vectors. Can use e.g. PCAthe set of motion vectors. Can use e.g. PCA
Remember:The optic flow field
•Vector field over the image:Vector field over the image:
[u,v] = f(x,y), u,v[u,v] = f(x,y), u,v = Vel vector, = Vel vector, x,y = x,y = Im posIm pos
•FOE, FOCFOE, FOC Focus of Expansion, Contraction Focus of Expansion, Contraction
Remember last lecture:
•Solving for the motion of a patchSolving for the motion of a patch
Over determined equation system:Over determined equation system:
ImImtt = Mu = Mu•Can be solved in e.g. least squares sense using Can be solved in e.g. least squares sense using matlab matlab u = M\Imu = M\Imtt
...à
@t@Im
...
0
@
1
A =
......
@x@Im
@y@Im
......
0
@
1
Aî xî y
!
t t+1
Know what type of motion(Greg Hager, Peter Belhumeur)
u’i = A ui + dE.g. Planar Object => Affine motion model:
It = g(pt, I0)
Mathematical Formulation
• Define a “warped image” Define a “warped image” gg– f(p,x) = x’f(p,x) = x’ (warping function), p warp parameters (warping function), p warp parameters– I(x,t)I(x,t) (image a location x at time t) (image a location x at time t)– g(p,Ig(p,Itt) = (I(f(p,x) = (I(f(p,x11),t), I(f(p,x),t), I(f(p,x22),t), … I(f(p,x),t), … I(f(p,xNN),t))’),t))’
• Define the Jacobian of warping functionDefine the Jacobian of warping function– M(p,t) =M(p,t) =
• ModelModel– II00 = g(p = g(ptt, I, It t )) (image I, variation model (image I, variation model gg, parameters , parameters pp))– I = I = MM(p(ptt, I, Itt) ) pp (local linearization (local linearization MM))
• Compute motion parametersCompute motion parameters p = (p = (MMT T MM))-1-1 MMT T I I where where MM = = MM(p(ptt,I,Itt))
@p@I
h i
Planar 3D motion
• From geometry we know that the correct From geometry we know that the correct plane-to-plane transform is plane-to-plane transform is
1.1. for a perspective camera the for a perspective camera the projective projective homographyhomography
2.2. for a linear camera (orthographic, weak-, para- for a linear camera (orthographic, weak-, para- perspective) the perspective) the affine warpaffine warp
uw
vw
" #
= Wa(p;a) =a3 a4
a5 a6
" #
p +a1
a2
" #
u0
v0
" #
= Wh(xh;h) = 1+h7u+h8v1
h1u h3v h5
h2u h4v h6
" #
Planar Texture Variability 1Affine Variability
•Affine warp functionAffine warp function
•Corresponding image variabilityCorresponding image variability
•Discretized for imagesDiscretized for images
= [B 1. . .B 6][y1; . . .;y6]T = Baya
É I a =P
i=16
@ai
@I wÉ ai = @u@I ;@v
@Iâ ã @a1
@u ááá @a6
@u
@a1
@v ááá @a6
@v
" # É a1...É a6
2
4
3
5
É I a = @u@I ;@v
@Iâ ã 1 0 ãu 0 ãv 00 1 0 ãu 0 ãv
" # y1...y6
2
4
3
5
uw
vw
" #
= Wa(p;a) =a3 a4
a5 a6
" #
p +a1
a2
" #
On The Structure of M
u’i = A ui + dPlanar Object + linear (infinite) camera
-> Affine motion model
X Y Rotation Scale Aspect Shear
M(p) = @g=@p
a3 a4
a5 a6
" #
= sR (Ê )a 00 1
" #1 0h 1
" #
Planar Texture Variability 2Projective Variability
• Homography warpHomography warp
• Projective variability:Projective variability:
• WhereWhere ,,
and and
u0
v0
" #
= Wh(xh;h) = 1+h7u+h8v1
h1u h3v h5
h2u h4v h6
" #
É I h = c1
1@u@I ;@v
@Iâ ã u 0 v 0 1 0 à c1
uc2 à c1
vc2
0 u 0 v 0 1 à c1
uc3 à c1
vc3
2
4
3
5É h1...É h8
2
4
3
5
c1 = 1+ h7u + h8vc3 = h2u + h4v+ h6
c2 = h1u + h3v+ h5
= [B 1. . .B 8][y1; . . .;y8]T = Bhyh
Planar motion under perspective projection
•Perspective plane-plane transforms defined by Perspective plane-plane transforms defined by homographieshomographies
Planar-perspective motion 3
• In practice hard to compute 8 parameter model In practice hard to compute 8 parameter model stably from one image, and impossible to find stably from one image, and impossible to find out-of plane variationout-of plane variation
•Estimate variability basis from several images:Estimate variability basis from several images:
Computed EstimatedComputed Estimated
Another idea Black, Fleet) Organizing flow fields
•Express flow field Express flow field f f in in subspace basis subspace basis mm
•Different “mixing” Different “mixing” coefficients coefficients aa correspond to different correspond to different motionsmotions
Mathematical formulation
Let:Let:
Mimimize objective function:Mimimize objective function:
==
WhereWhere
Motion basis
Robust error norm
ExperimentMoving camera
•4x4 pixel 4x4 pixel patchespatches
•Tree in Tree in foreground foreground separates well separates well
Summary
• Three types of visual motion extractionThree types of visual motion extraction1.1. Optic (image) flow: Find x,y – image velocitiesOptic (image) flow: Find x,y – image velocities
2.2. 3-6D motion: Find object pose change in image coordinates 3-6D motion: Find object pose change in image coordinates based more spatial derivatives (top down)based more spatial derivatives (top down)
3.3. Group flow vectors into global motion patterns (bottom up)Group flow vectors into global motion patterns (bottom up)
• Visual motion still not satisfactorily solved Visual motion still not satisfactorily solved problemproblem
Counterphase sin grating
•Spatio-temporal patternSpatio-temporal pattern– Time Time t, t, Spatial Spatial x,yx,y
s(x;y;t) = A cos(K xcosÊ + K ysinÊ à Ð)cos(! t)
Counterphase sin grating
• Spatio-temporal patternSpatio-temporal pattern– Time Time t, t, Spatial Spatial x,yx,y
Rewrite as dot product:Rewrite as dot product:
= + = +
s(x;y;t) = A cos(K xcosÊ + K ysinÊ à Ð)cos(! t)
21(cos([a;b] x
y
ô õà ! t) + cos([a;b] x
y
ô õ+ ! t)
Result: Standing wave is superposition of two moving waves
Analysis:
•Only one term: Motion left or rightOnly one term: Motion left or right
•Mixture of both: Standing waveMixture of both: Standing wave
•Direction can flip between left and rightDirection can flip between left and right
Severalmotion models
• Gradient: in Gradient: in Computer VisionComputer Vision
• Correlation: In bio Correlation: In bio visionvision
• Spatiotemporal Spatiotemporal filters: Unifying filters: Unifying modelmodel
Temporal response:
Adelson, Bergen ’85Adelson, Bergen ’85
Note: Terms from Note: Terms from
taylor of taylor of sin(t)sin(t)
Spatio-temporal Spatio-temporal D=DD=DssDDtt
Simplified:
•For our grating: For our grating: (Theta=0)(Theta=0)
•Write as sum of components:Write as sum of components:
= exp(…)*(acos… + bsin…)= exp(…)*(acos… + bsin…)
L s = 2A exp 2
à û2(kà K )2ð ñ
cos(þ à Ð)
Energy model:
•Sum odd and even phase Sum odd and even phase componentscomponents
•Quadrature rectifierQuadrature rectifier
Conclusion
•Evolutionary motion detection is importantEvolutionary motion detection is important
•Early processing modeled by Reichardt detector Early processing modeled by Reichardt detector or spatio-temporal filters.or spatio-temporal filters.
•Higher processing poorly understoodHigher processing poorly understood