Post on 23-Jul-2020
transcript
Lecture 6 -Stanford University
Lecture:RANSACandfeaturedetectors
JuanCarlosNiebles andRanjayKrishnaStanfordVisionandLearningLab
12-Oct-171
Lecture 6 -Stanford University
Whatwewilllearntoday?
• Amodelfittingmethodforedgedetection– RANSAC
• Localinvariantfeatures– Motivation– Requirements,invariances
• Keypoint localization– Harriscornerdetector
12-Oct-172
Lecture 6 -Stanford University
Whatwewilllearntoday
• Amodelfittingmethodforlinedetection– RANSAC
• Localinvariantfeatures– Motivation– Requirements,invariances
• Keypoint localization– Harriscornerdetector
12-Oct-173
Lecture 6 -Stanford University
FittingasSearchinParametricSpace
• Chooseaparametricmodeltorepresentasetoffeatures
• Membershipcriterionisnotlocal– Can’ttellwhetherapointbelongstoagivenmodeljustbylookingatthatpoint.
• Threemainquestions:– Whatmodelrepresentsthissetoffeaturesbest?– Whichofseveralmodelinstancesgetswhichfeature?– Howmanymodelinstancesarethere?
• Computationalcomplexityisimportant– Itisinfeasibletoexamineeverypossiblesetofparametersandeverypossiblecombinationoffeatures
Source:L.Lazebnik
12-Oct-174
Lecture 6 -Stanford University
Example:LineFitting• Whyfitlines?Manyobjectscharacterizedbypresenceofstraightlines
• Wait,whyaren’twedonejustbyrunningedgedetection?Slide credit: Kristen Grauman
12-Oct-175
Lecture 6 -Stanford University
DifficultyofLineFitting• Extraedgepoints(clutter),multiplemodels:– Whichpointsgowithwhichline,ifany?
• Onlysomepartsofeachlinedetected,andsomepartsaremissing:– Howtofindalinethatbridgesmissingevidence?
• Noiseinmeasurededgepoints,orientations:– Howtodetecttrueunderlyingparameters?
Slide credit: Kristen Grauman
12-Oct-176
Lecture 6 -Stanford University
Voting
• It’snotfeasibletocheckallcombinationsoffeaturesbyfittingamodeltoeachpossiblesubset.
• Votingisageneraltechniquewhereweletthefeaturesvoteforallmodelsthatarecompatiblewithit.– Cyclethroughfeatures,castvotesformodelparameters.– Lookformodelparametersthatreceivealotofvotes.
• Noise&clutterfeatureswillcastvotestoo,but typicallytheirvotesshouldbeinconsistentwiththemajorityof“good”features.
• Okifsomefeaturesnotobserved,asmodelcanspanmultiplefragments.
Slide credit: Kristen Grauman
12-Oct-177
Lecture 6 -Stanford University
RANSAC[Fischler &Bolles 1981]
• RANdom SAmple Consensus
• Approach:wewanttoavoidtheimpactofoutliers,solet’slookfor“inliers”,anduseonlythose.
• Intuition:ifanoutlierischosentocomputethecurrentfit,thentheresultinglinewon’thavemuchsupportfromrestofthepoints.
Slid
e cr
edit
: Kr
iste
n G
raum
an
12-Oct-178
Lecture 6 -Stanford University
RANSACloop:1. Randomlyselectaseedgroup ofpointsonwhichtobase
transformationestimate(e.g.,agroupofmatches)2. Computetransformationfromseedgroup3. Findinlierstothistransformation4. Ifthenumberofinliersissufficientlylarge,re-compute
least-squaresestimateoftransformationonalloftheinliers
• Keepthetransformationwiththelargestnumberofinliers
Slid
e cr
edit
: Kr
iste
n G
raum
an
12-Oct-179
RANSAC[Fischler &Bolles 1981]
Lecture 6 -Stanford University
RANSACLineFittingExample
• Task:Estimatethebestline– Howmanypointsdoweneedtoestimatetheline?
Slid
e cr
edit
: Ji
nxia
ngCh
ai
12-Oct-1710
Lecture 6 -Stanford University
RANSACLineFittingExample
• Task:Estimatethebestline
Sampletwopoints
Slid
e cr
edit
: Ji
nxia
ngCh
ai
12-Oct-1711
Lecture 6 -Stanford University
RANSACLineFittingExample
• Task:Estimatethebestline
Fitalinetothem
Slid
e cr
edit
: Ji
nxia
ngCh
ai
12-Oct-1712
Lecture 6 -Stanford University
RANSACLineFittingExample
• Task:Estimatethebestline
Total number of points within a threshold of line. Sl
ide
cred
it:
Jinx
iang
Chai
12-Oct-1713
Lecture 6 -Stanford University
RANSACLineFittingExample
• Task:Estimatethebestline
Total number of points within a threshold of line.
“7 inlier points”
Slid
e cr
edit
: Ji
nxia
ngCh
ai
12-Oct-1714
Lecture 6 -Stanford University
RANSACLineFittingExample
• Task:Estimatethebestline
Repeat,untilwegetagoodresult.
Slid
e cr
edit
: Ji
nxia
ngCh
ai
12-Oct-1715
Lecture 6 -Stanford University
RANSACLineFittingExample
• Task:Estimatethebestline
Repeat,untilwegetagoodresult.
“11 inlier points”
Slid
e cr
edit
: Ji
nxia
ngCh
ai
12-Oct-1716
Lecture 6 -Stanford University 12-Oct-1717
Lecture 6 -Stanford University
RANSAC:Howmanysamples?• Howmanysamplesareneeded?
– Supposew isfractionofinliers(pointsfromline).– n pointsneededtodefinehypothesis(2forlines)– k sampleschosen.
• Prob.thatasinglesampleofn pointsiscorrect:
• Prob.thatallk samplesfailis:
Þ Choosek highenoughtokeepthisbelowdesiredfailurerate.
nwknw )1( -
Slide credit: David Lowe
12-Oct-1718
Lecture 6 -Stanford University
RANSAC:Computedk(p=0.99)Sample
size
n
Proportion of outliers
5% 10% 20% 25% 30% 40% 50%
2 2 3 5 6 7 11 173 3 4 7 9 11 19 354 3 5 9 13 17 34 725 4 6 12 17 26 57 1466 4 7 16 24 37 97 2937 4 8 20 33 54 163 5888 5 9 26 44 78 272 1177
Slide credit: David Lowe
12-Oct-1719
Lecture 6 -Stanford University
AfterRANSAC• RANSACdividesdataintoinliersandoutliersandyieldsestimatecomputedfromminimalsetofinliers.
• Improvethisinitialestimatewithestimationoverallinliers(e.g.withstandardleast-squaresminimization).
• Butthismaychangeinliers,soalternatefittingwithre-classificationasinlier/outlier.
Slid
e cr
edit
: D
avid
Low
e
12-Oct-1720
Lecture 6 -Stanford University
RANSAC:ProsandCons
• Pros:– Generalmethodsuitedforawiderangeofmodelfittingproblems
– Easytoimplementandeasytocalculateitsfailurerate• Cons:
– Onlyhandlesamoderatepercentageofoutlierswithoutcostblowingup
– Manyrealproblemshavehighrateofoutliers(butsometimesselectivechoiceofrandomsubsetscanhelp)
• Avotingstrategy,TheHoughtransform,canhandlehighpercentageofoutliers
12-Oct-1721
Lecture 6 -Stanford University
Whatwewilllearntoday?
• Amodelfittingmethodforedgedetection– RANSAC
• Localinvariantfeatures– Motivation– Requirements,invariances
• Keypoint localization– Harriscornerdetector
12-Oct-1722
Somebackgroundreading:RickSzeliski,Chapter4.1.1;DavidLowe,IJCV2004
Lecture 6 -Stanford University
Imagematching:achallengingproblem
12-Oct-1723
Lecture 6 -Stanford University
by Diva Sian
by swashford
Slid
e cr
edit
: St
eve
Seit
z
12-Oct-1724
Imagematching:achallengingproblem
Lecture 6 -Stanford University
HarderCase
by Diva Sian by scgbt
Slid
e cr
edit
: St
eve
Seit
z
12-Oct-1725
Lecture 6 -Stanford University
HarderStill?
NASA Mars Rover images
Slid
e cr
edit
: St
eve
Seit
z
12-Oct-1726
Lecture 6 -Stanford University
AnswerBelow(Lookfortinycoloredsquares)
NASAMarsRoverimageswithSIFTfeaturematches(FigurebyNoahSnavely) Sl
ide
cred
it:
Stev
e Se
itz
12-Oct-1727
Lecture 6 -Stanford University
Motivationforusinglocalfeatures• Globalrepresentationshavemajorlimitations• Instead,describeandmatchonlylocalregions• Increasedrobustnessto
– Occlusions
– Articulation
– Intra-categoryvariations
θqφ
dq
φθ
d
12-Oct-1728
Lecture 6 -Stanford University
GeneralApproachN
pixe
ls
N pixels
Similarity measureAf
e.g. color
Bf
e.g. color
A1
A2 A3
Tffd BA <),(
1. Find a set of distinctive key-points
3. Extract and normalize the region content
2. Define a region around each keypoint
4. Compute a local descriptor from the normalized region
5. Match local descriptors
Slid
e cr
edit
: Ba
stia
n Le
ibe
12-Oct-1729
Lecture 6 -Stanford University
CommonRequirements• Problem1:
– Detectthesamepointindependently inbothimages
No chance to match!
We need a repeatable detector! Slid
e cr
edit
: D
arya
Fro
lova
, D
enis
Sim
akov
12-Oct-1730
Lecture 6 -Stanford University
CommonRequirements• Problem1:
– Detectthesamepointindependently inbothimages
• Problem2:– Foreachpointcorrectlyrecognizethecorrespondingone
We need a reliable and distinctive descriptor!
?
Slid
e cr
edit
: D
arya
Fro
lova
, D
enis
Sim
akov
12-Oct-1731
Lecture 6 -Stanford University
Invariance:GeometricTransformations
Slid
e cr
edit
: St
eve
Seit
z
12-Oct-1732
Lecture 6 -Stanford University
LevelsofGeometricInvariance
Slid
e cr
edit
: Ba
stia
n Le
ibe
12-Oct-1733
CS131 CS231a
Lecture 6 -Stanford University
Invariance:PhotometricTransformations
• Oftenmodeledasalineartransformation:– Scaling+Offset
Slid
e cr
edit
: Ti
nne
Tuyt
elaa
rs
12-Oct-1734
Lecture 6 -Stanford University
Requirements• Regionextractionneedstoberepeatable andaccurate
– Invarianttotranslation,rotation,scalechanges– Robust or covarianttoout-of-plane(»affine)transformations– Robust tolightingvariations,noise,blur,quantization
• Locality:Featuresarelocal,thereforerobusttoocclusionandclutter.
• Quantity:Weneedasufficientnumberofregionstocovertheobject.
• Distinctiveness:Theregionsshouldcontain“interesting”structure.
• Efficiency:Closetoreal-timeperformance.
Slid
e cr
edit
: Ba
stia
n Le
ibe
12-Oct-1735
Lecture 6 -Stanford University
ManyExistingDetectorsAvailable
• Hessian&Harris [Beaudet ‘78],[Harris‘88]
• Laplacian,DoG [Lindeberg ‘98],[Lowe‘99]
• Harris-/Hessian-Laplace [Mikolajczyk &Schmid ‘01]
• Harris-/Hessian-Affine [Mikolajczyk &Schmid ‘04]
• EBRandIBR [Tuytelaars &VanGool ‘04]
• MSER [Matas ‘02]
• SalientRegions [Kadir &Brady‘01]
• Others…
• ThosedetectorshavebecomeabasicbuildingblockformanyrecentapplicationsinComputerVision.
Slid
e cr
edit
: Ba
stia
n Le
ibe
12-Oct-1736
Lecture 6 -Stanford University
Whatwewilllearntoday?
• Amodelfittingmethodforedgedetection– RANSAC
• Localinvariantfeatures– Motivation– Requirements,invariances
• Keypoint localization– Harriscornerdetector
12-Oct-1737
Somebackgroundreading:RickSzeliski,Chapter4.1.1;DavidLowe,IJCV2004
Lecture 6 -Stanford University
Keypoint Localization
• Goals:– Repeatabledetection– Preciselocalization– InterestingcontentÞ Lookfortwo-dimensionalsignalchanges
Slid
e cr
edit
: Ba
stia
n Le
ibe
12-Oct-1738
Lecture 6 -Stanford University
FindingCorners
• Keyproperty:– Intheregionaroundacorner,imagegradienthastwoormoredominantdirections
• Cornersarerepeatable anddistinctive
C.Harris and M.Stephens. "A Combined Corner and Edge Detector.“Proceedings of the 4th Alvey Vision Conference, 1988.
Slid
e cr
edit
: Sv
etla
na L
azeb
nik
12-Oct-1739
Lecture 6 -Stanford University
CornersasDistinctiveInterestPoints• Designcriteria
– Weshouldeasilyrecognizethepointbylookingthroughasmallwindow(locality)
– Shiftingthewindowinany direction shouldgivealargechange inintensity(goodlocalization)
“edge”:no change along the edge direction
“corner”:significant change in all directions
“flat” region:no change in all directions Sl
ide
cred
it:
Alyo
sha
Efro
s
12-Oct-1740
Lecture 6 -Stanford University
Cornersversusedges
12-Oct-1741
Large
Large
Small
Large
Small
Small
Corner
Edge
Nothing
Lecture 6 -Stanford University
Cornersversusedges
12-Oct-1742
??
??Corner
Lecture 6 -Stanford University
HarrisDetectorFormulation
• Changeofintensityfortheshift[u,v]:
E(u,v) = w(x, y) I (x +u, y + v)− I (x, y)"# $%2
x ,y∑
IntensityShifted intensity
Window function
orWindow function w(x,y) =
Gaussian1 in window, 0 outside
Slid
e cr
edit
: Ri
ck S
zelis
ki
12-Oct-1743
Lecture 6 -Stanford University
HarrisDetectorFormulation• Thismeasureofchangecanbeapproximatedby:
whereM isa2´2matrixcomputedfromimagederivatives:
úû
ùêë
é»
vu
MvuvuE ][),(
M
Sumoverimageregion– theareawearecheckingforcorner
Gradient with respect to x, times gradient with respect to y
2
2,( , ) x x y
x y x y y
I I IM w x y
I I Ié ù
= ê úê úë û
å
Slid
e cr
edit
: Ri
ck S
zelis
ki
12-Oct-1744
Lecture 6 -Stanford University
HarrisDetectorFormulation
IxImage I IxIyIy
whereM isa2´2matrixcomputedfromimagederivatives:
M
Sumoverimageregion– theareawearecheckingforcorner
Gradient with respect to x, times gradient with respect to y
2
2,( , ) x x y
x y x y y
I I IM w x y
I I Ié ù
= ê úê úë û
å
Slid
e cr
edit
: Ri
ck S
zelis
ki
12-Oct-1745
Lecture 6 -Stanford University
WhatDoesThisMatrixReveal?• First,let’sconsideranaxis-alignedcorner:
• Thismeans:– Dominantgradientdirectionsalignwithx ory axis– Ifeitherλ iscloseto0,thenthisisnotacorner,solookforlocationswherebotharelarge.
• Whatifwehaveacornerthatisnotalignedwiththeimageaxes?
úû
ùêë
é=
úúû
ù
êêë
é=
åååå
2
12
2
00l
l
yyx
yxx
IIIIII
M
Slid
e cr
edit
: D
avid
Jac
obs
12-Oct-1746
Lecture 6 -Stanford University
WhatDoesThisMatrixReveal?• First,let’sconsideranaxis-alignedcorner:
• Thismeans:– Dominantgradientdirectionsalignwithx ory axis– Ifeitherλ iscloseto0,thenthisisnotacorner,solookforlocationswherebotharelarge.
• Whatifwehaveacornerthatisnotalignedwiththeimageaxes?
úû
ùêë
é=
úúû
ù
êêë
é=
åååå
2
12
2
00l
l
yyx
yxx
IIIIII
M
Slid
e cr
edit
: D
avid
Jac
obs
12-Oct-1747
Lecture 6 -Stanford University
GeneralCase
• SinceM issymmetric,wehave
• WecanvisualizeM asanellipsewithaxislengthsdeterminedbytheeigenvaluesandorientationdeterminedbyR
RRM úû
ùêë
é= -
2
11
00l
l
Direction of the slowest change
Direction of the fastest change
(lmax)-1/2
(lmin)-1/2
adap
ted
from
Dar
ya F
rolo
va,
Den
is S
imak
ov
(Eigenvalue decomposition)
12-Oct-1748
Lecture 6 -Stanford University
InterpretingtheEigenvalues• ClassificationofimagepointsusingeigenvaluesofM:
l1
“Corner”l1 andl2 arelarge,l1 ~ l2;E increasesinalldirections
l1 andl2 aresmall;E isalmostconstantinalldirections “Edge”
l1 >> l2
“Edge”l2 >> l1
“Flat”region
Slid
e cr
edit
: Kr
iste
n G
raum
an
l2
12-Oct-1749
Lecture 6 -Stanford University
CornerResponseFunction
• Fastapproximation– Avoidcomputingthe
eigenvalues– α:constant
(0.04to0.06)
l2
“Corner”θ > 0
“Edge”θ < 0
“Edge”θ < 0
“Flat”region
θ = det(M )−α trace(M )2 = λ1λ2 −α(λ1 +λ2 )2
l1Slid
e cr
edit
: Kr
iste
n G
raum
an
12-Oct-1750
Lecture 6 -Stanford University
WindowFunctionw(x,y)
• Option1:uniformwindow– Sumoversquarewindow
– Problem:notrotationinvariant
• Option2:SmoothwithGaussian– Gaussianalreadyperformsweightedsum
– Resultisrotationinvariant
1 in window, 0 outside
2
2,( , ) x x y
x y x y y
I I IM w x y
I I Ié ù
= ê úê úë û
å
Gaussian
2
2,
x x y
x y x y y
I I IM
I I Ié ù
= ê úê úë û
å
2
2( ) x x y
x y y
I I IM g
I I Is
é ù= *ê ú
ê úë û
Slid
e cr
edit
: Ba
stia
n Le
ibe
12-Oct-1751
Lecture 6 -Stanford University
Summary:HarrisDetector[Harris88]
• Computesecondmomentmatrix(autocorrelationmatrix)
1. Image derivatives
Ix Iy
2. Square of derivatives
Ix2 Iy2 IxIy
3. Gaussian filter g(sI) g(Ix2) g(Iy2) g(IxIy)
R
2
2
( ) ( )( , ) ( )
( ) ( )x D x y D
I D Ix y D y D
I I IM g
I I Is s
s s ss s
é ù= *ê ú
ê úë û
2 2 2 2 2 2( ) ( ) [ ( )] [ ( ) ( )]x y x y x yg I g I g I I g I g Ia= - - +
θ = det[M (σ I ,σ D )]−α[trace(M (σ I ,σ D ))]2
4. Cornerness function – two strong eigenvalues
5. Perform non-maximum suppression
Slid
e cr
edit
: Kr
ysti
anM
ikol
ajcz
yk
12-Oct-1752
Lecture 6 -Stanford University
HarrisDetector:Workflow
Slid
e ad
apte
d fr
om D
arya
Fro
lova
, D
enis
Sim
akov
12-Oct-1753
Lecture 6 -Stanford University
HarrisDetector:Workflow- computercornerresponsesθ
Slid
e ad
apte
d fr
om D
arya
Fro
lova
, D
enis
Sim
akov
12-Oct-1754
Lecture 6 -Stanford University
Slid
e ad
apte
d fr
om D
arya
Fro
lova
, D
enis
Sim
akov
HarrisDetector:Workflow- Takeonlythelocalmaximaofθ,whereθ>threshold
12-Oct-1755
Lecture 6 -Stanford University
HarrisDetector:Workflow- ResultingHarrispoints
Slid
e ad
apte
d fr
om D
arya
Fro
lova
, D
enis
Sim
akov
12-Oct-1756
Lecture 6 -Stanford University
HarrisDetector– Responses[Harris88]
Effect: A very precise corner detector.
Slid
e cr
edit
: Kr
ysti
anM
ikol
ajcz
yk
12-Oct-1757
Lecture 6 -Stanford University
HarrisDetector– Responses[Harris88]
Slid
e cr
edit
: Kr
ysti
anM
ikol
ajcz
yk
12-Oct-1758
Lecture 6 -Stanford University
HarrisDetector– Responses[Harris88]
Slid
e cr
edit
: Kr
iste
n G
raum
an
• Resultsarewellsuitedforfindingstereocorrespondences
12-Oct-1759
Lecture 6 -Stanford University
HarrisDetector:Properties
• Translationinvariance?
Slid
e cr
edit
: Kr
iste
n G
raum
an
12-Oct-1760
Lecture 6 -Stanford University
HarrisDetector:Properties
• Translationinvariance• Rotationinvariance?
Ellipse rotates but its shape (i.e. eigenvalues) remains the same
Corner response θ is invariant to image rotation
Slid
e cr
edit
: Kr
iste
n G
raum
an
12-Oct-1761
Lecture 6 -Stanford University
HarrisDetector:Properties
• Translationinvariance• Rotationinvariance• Scaleinvariance?
Not invariant to image scale!
All points will be classified as edges!
Corner
Slid
e cr
edit
: Kr
iste
n G
raum
an
12-Oct-1762
Lecture 6 -Stanford University
Whatwearelearnedtoday?
• Amodelfittingmethodforedgedetection– RANSAC
• Localinvariantfeatures– Motivation– Requirements,invariances
• Keypoint localization– Harriscornerdetector
12-Oct-1763