Optical Flow

CS 510 Lecture #21

April 11h, 2014

Where are we?

•  We know how to implement a BoW •  BoW was invented to match still images •  We can update the feature extraction step

to find STIPs (spatio-temporal IPs) – 3D corners (Laptev) – Gabor peaks (Dollár)

•  But if we can’t describe motion…

4/16/14   CS  510,  Image  Computa4on,  ©Ross  Beveridge  &  Bruce  Draper   2  

Describing Motion : Flow

•  Motion is best described as the 2D motion of surface points over time – Find an easy-to-recognize point on object – Record its (x1,y1) position at time T1

– Record its (x2,y2) position at time T2

–  Its flow vector (dx/dt, dy/dt) is (x2-x1, y2-y1) •  Of course, the devil is in the details

4/16/14   CS  510,  Image  Computa4on,  ©Ross  Beveridge  &  Bruce  Draper   3  

Example of Sparse Flow

4/16/14   CS  510,  Image  Computa4on,  ©Ross  Beveridge  &  Bruce  Draper   4  

hDp://docs.opencv.org/trunk/doc/py_tutorials/py_video/py_lucas_kanade/py_lucas_kanade.html  

Computing Sparse Flow (simple) •  Pick good points to track

– Point matching must be spatially precise •  Frame to frame movements are small •  Corners (structure tensors) were invented for this

purpose – Patterns must be unique

•  Repeated patterns must be avoided •  Checkerboards, brick walls, etc. should be avoided

– OpenCV has a goodFeaturesToTrack function •  Use image around point as template

– Find position of best match in next image – Compute flow from position of match

4/16/14   CS  510,  Image  Computa4on,  ©Ross  Beveridge  &  Bruce  Draper   5  

Sparse Flow Issues

4/16/14   CS  510,  Image  Computa4on,  ©Ross  Beveridge  &  Bruce  Draper   6  

Example of Dense Flow •  Scene from PETS

video •  Farneback 2003 •  From OpenCV

4/16/14   CS  510,  Image  Computa4on,  ©Ross  Beveridge  &  Bruce  Draper   7  

Focus of Expansion (FoE)

•  If the camera is moving into the scene, all flow field vectors point away from the point the camera is moving toward

•  This point is called the “focus of expansion”

•  Simple form of navigation (insects use it)

4/16/14   CS  510,  Image  Computa4on,  ©Ross  Beveridge  &  Bruce  Draper   8  

hDp://w

ww.hizook.com

/blog/2010/02/16/learning-­‐es4mate-­‐robot-­‐m

o4on-­‐and-­‐find-­‐unexpected-­‐objects-­‐op4cal-­‐flow  

Modeling Camera Motion

4/16/14   CS  510,  Image  Computa4on,  ©Ross  Beveridge  &  Bruce  Draper   9  

Which  way  is  the  camera  moving  in  these  scenes?  

hDp://w

ww.hizook.com

/blog/2010/02/16/learning-­‐es4mate-­‐robot-­‐m

o4on-­‐and-­‐find-­‐unexpected-­‐objects-­‐op4cal-­‐flow