Date post: | 16-Jan-2016 |
Category: |
Documents |
Upload: | violet-gilmore |
View: | 212 times |
Download: | 0 times |
Characterizing activity in video shots based on
salient points
Nicolas Moënne-LoccozViper groupComputer vision & multimedia laboratory University of Geneva
NML - CVML - UniGe 2
Outline
• Context
• Video Activity extraction– Spatial salient points– Spatio-temporal salient points– Spatio-temporal salient regions
• Results
• Conclusion
NML - CVML - UniGe 3
Context• Describe visual content of video
Index, retrieve and browse video database
• Requirements– Generic approach (v.s. domain oriented)– Local approach (v.s. global description of the content)– Computationally efficient approach
• Video activity : salient region of the video 3D space
NML - CVML - UniGe 4
Context0 50 70Frames
Activity #1
Activity #2
Description in space and time of video activity Inference based on video object and event relationships High level indexing
NML - CVML - UniGe 5
Context
• Related approaches : Spatio-temporal segmentation– Segmentation problem– Computational efficiency
• Our approach :– Spatio-temporal salient points– Spatial grouping of salient points– Temporal matching of salient regions
Set of activities
NML - CVML - UniGe 6
Overview
Salient points& trajectories
Global motionestimation
Motion outliers
Spatial grouping
Video stream
Salient extraction
Temporal matching
Salient extraction
NML - CVML - UniGe 7
Salient points
• Points in the image space– Repetitive (robust)– High information content
Scale invariant interest points (Mikolajczyk, Schmid 2001)
– One of the most robust– Salient points with characteristic scale
NML - CVML - UniGe 8
Salient point extraction
• Linear Scale-Space :
• Harris function :
• Salient points (image space) : local maxima h(v,s)
• Laplacian over scale :
• Salient points (scale space) : local maxima l(v,s) & h(v,s)
H žv , s Ÿ= s 2G žv , ? ŸL x
2 žv , s Ÿ L x L yžv , sŸ
L x L yžv , sŸ L x2 žv , sŸ
NML - CVML - UniGe 9
Salient point extraction
• Example :
scale
NML - CVML - UniGe 10
Salient point extraction
scale
NML - CVML - UniGe 11
Motion estimation
• Goal :– Find points having salient temporal behaviour
Estimate background motion model Select points that do not follow this background motion
model
• Estimation :– Compute salient point trajectories– Estimate corresponding affine motion model
NML - CVML - UniGe 12
Trajectories
• Point descriptors : Local Grayvalue Invariants
• Point distance : Mahalanobis distance
NML - CVML - UniGe 13
Trajectories
• Goodness of match :
• Candidate matching points– Matches with spatial distance below a threshold
• Relaxation process :– Disambiguating set of candidate matches– Greedy Winner-Takes-All algorithm
NML - CVML - UniGe 14
Motion estimation
• Affine motion model :
• Estimate model from trajectories– Iterative least square error estimate (Tukey M-Estimator)
select points that belong to the global motion model
Assumption : +50% points belong to the background
NML - CVML - UniGe 15
Motion estimation
Points of the background and their motion estimated using the presented approach
All points and their motion estimated by a dense motion estimator
NML - CVML - UniGe 16
Spatio-temporal salient points
• Points whose trajectory does not fit the global motion model
Outliers (moving objects)
• Points without trajectory (no matching point) New points (appearing or deformable objects)
NML - CVML - UniGe 17
Spatio-temporal salient points
Fixed camera Moving camera
NML - CVML - UniGe 18
Salient regions
• Set of spatio-temporal salient points
Feature distribution of points (RGB colour features) Spatial distribution of points
• Grouping process : Estimate salient region models
NML - CVML - UniGe 19
Feature model
• Feature description– A salient point is characterized by the feature distribution of its
neighbourhood
– Assumption : maximum of four regions in the neighbourhood of the points
– Compute the corresponding colour distributions :• K-means clustering• Gaussian model
• Gaussian models clustering– Greedy algorithm (AHC)
Set of Gaussian distributions representing the distribution of the neighbourhood of the salient points :
NML - CVML - UniGe 20
Salient region model
• Feature models– Mixture of Gaussians
Corresponding weight of each Gaussian
• Spatial model : – Estimate spatial pdf from salient points & associated scale
NML - CVML - UniGe 21
Salient region models
• Iterate a RanSaC algorithm
• Estimate salient region model– Robust estimation (Tukey M-estimator)– Cost function :
NML - CVML - UniGe 22
Salient regions
Fixed camera Moving camera
NML - CVML - UniGe 23
Temporal matching
• Spatio-temporal salient regions of arbitrary length
Matching of salient regions
• Use salient points trajectories
1. Match regions with the highest number of matching points
NML - CVML - UniGe 24
Results - Meetings
NML - CVML - UniGe 25
Results – Misc
NML - CVML - UniGe 26
Conclusion
• Contribution– Highly informational content descriptor– Generic content descriptor– Local in space and time content descriptor
• Limitation– Noisy & short activity
• Ongoing work– Temporal filtering of activity– Indexing of videos through the set of activity