Frame Decimation Frame Decimation for Structure and for Structure and

MotionMotion

Young Ki BaikYoung Ki BaikCV Lab.CV Lab.

ContentsContents

MotivationMotivation

Frame Selection methodFrame Selection method

Conclusion and Future worksConclusion and Future works

MotivationMotivation

Profit from video sequenceProfit from video sequence CorrespondenceCorrespondence

To save large number of inlier correspondence from small To save large number of inlier correspondence from small motionmotion

X

O

MotivationMotivation

Problem #1Problem #1 Different amount of motionDifferent amount of motion

Fairly small motion to allow automatic matchingFairly small motion to allow automatic matching Significant parallax and large baseline to assure of Significant parallax and large baseline to assure of

a well-condition for 3d reconstruction a well-condition for 3d reconstruction

Uncertainty

boundary

Small base line

Large base line

MotivationMotivation

Problem #2Problem #2 Unsharp frames by Bad focusUnsharp frames by Bad focus

Giving us ill-condition of corner detection.Giving us ill-condition of corner detection. Eliminating unsharp frame to save corner info.Eliminating unsharp frame to save corner info.

MotivationMotivation

Solution → Frame SelectionSolution → Frame Selection Deleting redundant and unsharp framesDeleting redundant and unsharp frames Isotropic motionIsotropic motion

MotivationMotivation

Frame SelectionFrame Selection Frame decimation for structure and motionFrame decimation for structure and motion

David Nister, SMILE2000David Nister, SMILE2000 An assessment of information criteria for motion moAn assessment of information criteria for motion mo

del selectiondel selection Torr, CVPR97Torr, CVPR97

Key frame Selection for Camera Motion and StructurKey frame Selection for Camera Motion and Structure Estimation from Multiple Viewse Estimation from Multiple Views

Thormahlen, ECCV2004Thormahlen, ECCV2004

Frame SelectionFrame Selection

Sharpness measureSharpness measure For removing unsharp framesFor removing unsharp frames

Shot boundary detectionShot boundary detection For selecting subset of framesFor selecting subset of frames

Selection of a subsequence of framesSelection of a subsequence of frames For eliminating redundant frames For eliminating redundant frames

Frame SelectionFrame Selection

Sharpness Measure Sharpness Measure for saving corner informationfor saving corner information The measure of image sharpnessThe measure of image sharpness

The mean square of the horizontal and vertical The mean square of the horizontal and vertical derivatives.derivatives.

22

),(

)1,()1,(),1(),1()(#2

1)(

yxfyxfyxfyxfI

ISIyx

Frame SelectionFrame Selection

Sharpness Measure Sharpness Measure Only to consider the relative sharpness of Only to consider the relative sharpness of

similar imagessimilar images

Sudden

Changes

Frame SelectionFrame Selection

Shot Boundary Detection Shot Boundary Detection Shot boundaryShot boundary

These occur when the camera has been turned off These occur when the camera has been turned off and then turned on again at a new place.and then turned on again at a new place.

Shot boundary

Frame SelectionFrame Selection

Shot Boundary Detection Shot Boundary Detection Shot boundaries are detected by evaluating the correlaShot boundaries are detected by evaluating the correla

tion between adjacent frames after global motion comtion between adjacent frames after global motion compensation.pensation.

Mean of Normalized correlation value (TMean of Normalized correlation value (Tsbsb = 0.75) = 0.75)

Shot boundary detection

0.9

0.5

0.9

valueIntensityI

HomographyH

ROI

IIR

)()()(#

1 2

)(

1112

x

xHx

Frame SelectionFrame Selection

Selection of Subsequence Selection of Subsequence For deleting redundant framesFor deleting redundant frames

Considering two propertiesConsidering two properties Normalized Correlation (NC) constraintNormalized Correlation (NC) constraint Distant constraintDistant constraint

Frame SelectionFrame Selection

Normalized Correlation (NC) constraintNormalized Correlation (NC) constraint Check the mean of NC value between near frames.Check the mean of NC value between near frames. Delete Delete FFii when the mean of NC value between when the mean of NC value between FFi-1i-1 and and

FFi+1i+1 is bigger than T(=0.95). is bigger than T(=0.95).

1iF 1iF

iF

95.0T

Frame SelectionFrame Selection

Distance constraintDistance constraint Check maximum distance in correspondences.Check maximum distance in correspondences. Delete Delete FFii when maximum distance is smaller than when maximum distance is smaller than

TTdd (= image size/10). (= image size/10).

Maximum distance of correspondence

1iF 1iF

iF

Frame SelectionFrame Selection

Selection of SubsequenceSelection of Subsequence Deleting process is repeated until no Deleting process is repeated until no

additional deletions occur.additional deletions occur.

Conclusion and Future Conclusion and Future worksworks

ConclusionConclusion Method of the frame selection in video Method of the frame selection in video

sequencesequence

Future worksFuture works Usage of video sequenceUsage of video sequence ImplementationImplementation