An Update on Features - SURF, BRISK, ORB and FREAK · SURF boxfilterpyramid Haarwavelets binary...

AN UPDATE ON FEATURES

SURF, BRISK, ORB and FREAK

Stefan Haller ↦ [email protected]

Dresden, 12th January 2015

Outline

1. Introduction to Features

2. SURF – Speeded Up Robust Features

3. BRISK – Binary Robust Invariant Scalable Keypoints

4. ORB – Oriented FAST and Rotated BRIEF

5. FREAK – Fast Retina Keypoints

6. Comparison

7. References

8. Discussion

TU Dresden, 12th January 2015 An Update on Features Folie 2 von 51

Introduction to Features


Image Featuresachievement :

• detect image features: unique interest points

• describe image feature: sample region of image patcharound point

applications include:

• object recognition and tracking

• image matching and stitching

• robotic mapping and 3D modeling

important :

• repeatability and reliability

• good performance (description, matching)

• speed


1st Step: Feature Detection

By Retardo(http://en.wikipedia.org/wiki/File:Corner.png)[Public domain], viaWikimedia Commons


2nd Step: Feature Description

Bay, H., Tuytelaars, T., & VanGool, L. (2006). Surf: Speededup robust features. InComputer Vision–ECCV2006 (pp. 404-417). SpringerBerlin Heidelberg.

Lowe, D. G. (2004).Distinctive image featuresfrom scale-invariantkeypoints. Internationaljournal of computer vision,60(2), 91-110.


History

Traditional (slower, accurate):

1999 Scale Invariant Feature Transform (Lowe)

2006 Speeded Up Robust Features (Bay, Tuytelaars, Van Gool)

Binary (faster, real time, smartphone, performance):

2010 Binary Robust Independent Elementary Features(Michael Calonder, et al.)

2011 Oriented FAST and Rotated BRIEF (Ethan Rublee et al.)

2011 Binary Robust Invariant Scalable Keypoints(Leutenegger, Chli, Siegwart)

2012 Fast Retina Keypoint (Alahi, Ortiz, Vandergheynst)


History

Traditional (slower, accurate):

1999 Scale Invariant Feature Transform (Lowe)

2006 Speeded Up Robust Features (Bay, Tuytelaars, Van Gool)

Binary (faster, real time, smartphone, performance):

2010 Binary Robust Independent Elementary Features(Michael Calonder, et al.)

2011 Oriented FAST and Rotated BRIEF (Ethan Rublee et al.)

2011 Binary Robust Invariant Scalable Keypoints(Leutenegger, Chli, Siegwart)

2012 Fast Retina Keypoint (Alahi, Ortiz, Vandergheynst)


SURF – Speeded UpRobust Features


Properties

• scale and rotation invariant (Upright SURF only scaleinvariant)

• faster computation and faster matching

• achieved by integral images and smaller feature vector size

• ideas of SIFT, but much more simplified


Feature Detection: Fast-Hessian• based on Hessian matrix (Laplacian of Gaussian, LoG)• reminder: LoG approximated by Difference of Gaussian,

DoG• SURF uses “Fast-Hessian Detector”: approximation withbox filters







box filters→very fast for arbitrary size (integral imagemethod)

Bay, H., Tuytelaars, T., & Van Gool, L. (2006). Surf: Speeded up robust features. In ComputerVision–ECCV 2006 (pp. 404-417). Springer Berlin Heidelberg.


Feature Detection: Scale invariance

• use of scale space pyramid

• box filters allow to blur base image of current octave

• parallel creation of scales for octave possible!

Lowe, D. G. (2004).Distinctive image featuresfrom scale-invariantkeypoints. Internationaljournal of computer vision,60(2), 91-110.


Feature Description: OrientationAssignment

• (step is skipped for USURF)• consider circular region around image patch (size

dependes on scale)• calculate Haar wavelet responses in 𝑥 and 𝑦 direction

(integral images → 6 ops for any sampling point)

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

0 1

“Haar wavelet”. Licensed under CC BY-SA 3.0 viaWikimedia Commons -http://commons.wikimedia.org/wiki/File:Haar_wavelet.svg

• representation in vector space• sliding window → sum up responses → choose largestvector


Feature Description: Dominant Direction

verticalresponse

horizontalresponse



verticalresponse

horizontalresponse



verticalresponse

horizontalresponse



verticalresponse

horizontalresponse



verticalresponse

horizontalresponse



verticalresponse

horizontalresponse



verticalresponse

horizontalresponse


Feature Description Vector

• square region around IP (rotated)

• split into 4 × 4 subregion

• for each subregion: 5 × 5 regularly spaced sample points

• compute Haar wavelet response horiz. and vert.

• for each: sum of d𝑥, d𝑦, |d𝑥|, |d𝑦|

→ 4 × 4 region, respectively 4 dim. vector → 64 dim. vector


Feature Description Vector

Bay, H., Tuytelaars, T., & Van Gool, L. (2006). Surf: Speeded up robust features. In ComputerVision–ECCV 2006 (pp. 404-417). Springer Berlin Heidelberg.


BRISK – Binary RobustInvariant Scalable

Keypoints


Properties

• dramatically lower computational complexity

• assembly of bit-string vector (no gradients, etc!)

• general property of binary descriptors: use of Hammingdistance instead of Euclidean distance

• Hamming distance ≙ XOR both vectors and count “1”s→ boosted by SSE/AVX

• faster than SIFT/SURF, comparable performance

(binary descriptors) especially suited for:

• real time requirements

• low power devices


Scale-Space Keypoint Detection

• extension of AGAST which is itself extension FAST

• invariant of scale

• estimates keypoint scale in continuous scale-space


Keypoint Description

• composed of binary string, simple brightness comparisontests

• neighborhood sampling: deterministic, equally spaced,concentric circles

Leutenegger, S., Chli, M., & Siegwart,R. Y. (2011, November). BRISK:

Binary robust invariant scalablekeypoints. In Computer Vision(ICCV), 2011 IEEE International

Conference on (pp. 2548-2555). IEEE.


Long and Short Pairs𝒜 = {(𝐩𝑖, 𝐩𝑗) ∈ ℝ2 × ℝ2 ∣ 𝑖, 𝑗 ∈ ℕ, 𝑗 < 𝑖 < 𝑁}

𝒮 = {(𝐩𝑖, 𝐩𝑗) ∈ 𝒜 ∣ ‖𝐩𝑗 − 𝐩𝑖‖ < 𝛿𝑚𝑎𝑥} ⊆ 𝒜ℒ = {(𝐩𝑖, 𝐩𝑗) ∈ 𝒜 ∣ ‖𝐩𝑗 − 𝐩𝑖‖ > 𝛿𝑚𝑖𝑛} ⊆ 𝒜

Leutenegger, S., Chli, M., & Siegwart,R. Y. (2011, November). BRISK:

Binary robust invariant scalablekeypoints. In Computer Vision(ICCV), 2011 IEEE International

Conference on (pp. 2548-2555). IEEE.


Orientation Computation

• long distance pairs used• calculate local gradient between pairs• sum up all local gradients

𝐠(𝐩𝑖, 𝐩𝑗) = (𝐩𝑗 − 𝐩𝑖) ⋅ 𝐼(𝐩𝑗, 𝜎𝑗) − 𝐼(𝐩𝑖, 𝜎𝑖)‖𝐩𝑗 − 𝐩𝑖‖2

𝐠 = (𝑔𝑥𝑔𝑦

) = 1𝐿 ⋅ ∑

(𝐩𝑖,𝐩𝑗)∈ℒ𝐠(𝑝𝑖, 𝑝𝑗)

𝜃 = arctan2(𝑔𝑦, 𝑔𝑥)TU Dresden, 12th January 2015 An Update on Features Folie 21 von 51

Building the Descriptor

• binary descriptor: comparison of points with binaryoutcome

• short distance pairs used

𝑏 = {1 if 𝐼(𝐩𝛼𝑗 , 𝜎𝑗) > 𝐼(𝐩𝛼

𝑖 , 𝜎𝑖)0 otherwise

∀(𝐩𝛼𝑖 , 𝐩𝛼

𝑗 ) ∈ 𝒮

• Hamming distance for matching

• XOR two vectors and count resulting “1”s


ORB – Oriented FASTand Rotated BRIEF


Properties

• fast, computational efficient

• real time and low power device

• (all other properties of binary descriptors)


Detector (oriented FAST)

• modified FAST: oriented FAST

• measures intensity between center pixel and those incircular ring around center

shortcomings of FAST approach:

1. no quality measure (“cornerness”)solution: use Harris cornerness measure

2. not scale invariantsolution: use scale pyramid (like SIFT)

3. not rotation invariantsolution: calculate intensity centroid


Intensity Centroid (Rotation Invariance)

assumption: corner’s intensity is offset from center

∀ 𝑝, 𝑞 ∈ {0, 1}⏟⏟⏟⏟⏟binary selector

for x and ydirection

∶ 𝑚𝑝𝑞 = ∑𝑥,𝑦⏟

circularwindow

𝑥𝑝 𝑦𝑞⏟weighted

bycoordinate

𝐼(𝑥, 𝑦)⏟image

function

𝐶 = (𝑚10𝑚00

, 𝑚01𝑚00

)

dominant direction ⃗⃗⃗⃗ ⃗⃗ ⃗⃗⃗ ⃗⃗ ⃗𝑂𝐶 : 𝜃 = atan2(𝑚01, 𝑚10)


Intensity Weighted Center of Mass


Descriptor (modified BRIEF)

• binary descriptor, no sampling pattern

• 𝜏(𝑝; 𝑥, 𝑦) = {1 if 𝑝(𝑥) ≥ 𝑝(𝑦)0 otherwise

• feature vector 𝑓𝑛(𝑝) ≔ ∑1≤𝑖≤𝑛

2𝑖−1 𝜏(𝑝; 𝑥𝑖, 𝑦𝑖)

• vector length 𝑛 = 256 bit


Learning of Pairs

• 256 bit, but thousands of possible pairs

• Which pairs provide best information for given patch?

• We want to maximize variance and thus minimizecorrelation.

• Paper suggests learning algorithm

• Input: reference image series / Output: static list of pairs

⌛ Not enough time to explain in detail→ Discussion afterwards?


FREAK – Fast RetinaKeypoints


Properties and Motivation

• binary descriptor, no feature detector specified

• inspired by human visual system, precisely retina

• at same time enforcing low computational complexity

• tries to be faster and more robust than SIFT, SURF orBRISK

• (all other properties of binary descriptors)


Biological Background• human retina extracts details from images using DoGs• differences are encoded into action potentials• several photo-receptors influence ganglion cell

Alahi, A., Ortiz, R., &Vandergheynst, P. (2012,June). Freak: Fast retinakeypoint. In ComputerVision and PatternRecognition (CVPR),2012 IEEE Conferenceon (pp. 510-517). Ieee.


Biological Background• human retina extracts details from images using DoGs• differences are encoded into action potentials• several photo-receptors influence ganglion cell• size and dendritic field increases with radial distance

from fovea• spatial distribution of ganglion cells reduces exponentially

Alahi, A., Ortiz, R., &Vandergheynst, P. (2012,June). Freak: Fast retinakeypoint. In ComputerVision and PatternRecognition (CVPR), 2012IEEE Conference on (pp.510-517). Ieee.


FREAK’s Sampling Pattern

• sampling pattern similar to retinal ganglion cells

• receptive fields are overlapping → redundancy

• increases performance, better discriminative power

Alahi, A., Ortiz, R., & Vandergheynst,P. (2012, June). Freak: Fast retina

keypoint. In Computer Vision andPattern Recognition (CVPR), 2012 IEEE

Conference on (pp. 510-517). Ieee.


FREAK’s methods

⌛ skipped:

orientation: based on BRISK

sampling pairs: learningalgorithm based on ORB

the results are quite interesting:coarse-to-fine ordering isautomatically preferred!

Alahi, A., Ortiz, R., & Vandergheynst, P. (2012,June). Freak: Fast retina keypoint. In ComputerVision and Pattern Recognition (CVPR), 2012 IEEEConference on (pp. 510-517). Ieee.


Saccadic Search• eyes perform discontinuous individual movements

(“saccades”)

• fovea captures high-resolution → high density• critical role during recognition and matching

• perifoveal area captures less detailed information• used to compile first estimates

“Szakkad” by Original file: SpooSpa. Derivative:Simon Viktória - Derivative work from File:Faceof SpooSpa.jpg. Licensed under CC BY-SA 2.0 viaWikimedia Commons -http://commons.wikimedia.org/wiki/File:Szakkad.jpg


Saccadic Search

• FREAK performs several steps

• at first parse first 16 byte of descriptor → corresponds toouter coarse information

• ≥ 90% of candidates discarded with first 16 bytes

• use SIMD: comparing 16 byte as fast as 1 byte


Comparison


Summary

real valued detector descriptorSIFT LoG pyramid gradient valuesSURF box filter pyramid Haar wavelets

binary detector sample pattern orientation sampling pairsBRISK AGAST concentric circles gradient short pairsORB oFAST none moments learnedFREAK N/A overlapping con-

centric circlesgradient (pre-selected pairs)

learned


Performance Evaluation

• source: “Evaluation of Local Detectors and Descriptors forFast Feature Matching”

• comparison of matching speed and precision/accuracy

• only publicly available implementations can be tested(→ OpenCV)


Implementation Details

• matching: nearest neighborhood search (NN)1. kd-trees (geometric, partition of dimensions)

→ for real-valued descriptors

2. hashing (project similar data into same bucket, Hamming space)→ for binary descriptors

• ϵ-ANN: approximated NN

• image data set: standard graffiti image sequences

• 24× 3.47 GHz, 12 MB cache, Intel Xeon X5690, 99 GBRAM, x64 Ubuntu 10.04

• SSE 4.2 enabled (fast POPCNT)

• algorithms used with default parameters (except FAST)


Miksik, O., & Mikolajczyk, K. (2012, November). Evaluation of local detectors and descriptors forfast feature matching. In Pattern Recognition (ICPR), 2012 21st International Conference on (pp.2681-2684). IEEE.






Performance Measures

recall = #true positive#true positive + #false negative

precision = #true positive#true positive + #false positive




My Conclusion

• SIFT: slow + good quality

• BRISK: fast + good quality

• FREAK ≈ BRISK

• ORB: faster + littlebit less quality


References


References (Primary)

📖 Bay, H., Tuytelaars, T., & Van Gool, L. (2006). Surf: Speeded up robust features. InComputer Vision–ECCV 2006 (pp. 404-417). Springer Berlin Heidelberg.

📖 Rublee, E., Rabaud, V., Konolige, K., & Bradski, G. (2011, November). ORB: an efficientalternative to SIFT or SURF. In Computer Vision (ICCV), 2011 IEEE InternationalConference on (pp. 2564-2571). IEEE.

📖 Leutenegger, S., Chli, M., & Siegwart, R. Y. (2011, November). BRISK: Binary robustinvariant scalable keypoints. In Computer Vision (ICCV), 2011 IEEE InternationalConference on (pp. 2548-2555). IEEE.

📖 Alahi, A., Ortiz, R., & Vandergheynst, P. (2012, June). Freak: Fast retina keypoint. InComputer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on (pp. 510-517).Ieee.

📖 Miksik, O., & Mikolajczyk, K. (2012, November). Evaluation of local detectors anddescriptors for fast feature matching. In Pattern Recognition (ICPR), 2012 21stInternational Conference on (pp. 2681-2684). IEEE.


References (Secondary)

📖 Lowe, D. G. (2004). Distinctive image features from scale-invariant keypoints. Internationaljournal of computer vision, 60(2), 91-110.

📖 Rosten, E., & Drummond, T. (2006). Machine learning for high-speed corner detection. InComputer Vision–ECCV 2006 (pp. 430-443). Springer Berlin Heidelberg.

📖 Juan, L., & Gwun, O. (2009). A comparison of sift, pca-sift and surf. International Journal ofImage Processing (IJIP), 3(4), 143-152.

📖 Calonder, M., Lepetit, V., Strecha, C., & Fua, P. (2010). Brief: Binary robust independentelementary features. In Computer Vision–ECCV 2010 (pp. 778-792). Springer BerlinHeidelberg.

📖 Bekele, D., Teutsch, M., & Schuchert, T. (2013). Evaluation of binary keypoint descriptors.In Image Processing (ICIP), 2013 20th IEEE International Conference (pp. 3652-3656). doi:10.1109/ICIP.2013.6738753

📖 Figat, J., Kornuta, T., & Kasprzak, W. (2014). Performance Evaluation of Binary Descriptorsof Local Features. In Computer Vision and Graphics (pp. 187-194). Springer InternationalPublishing.


Discussion


Discussion

• real valued vs. binary descriptor

• simplicity vs. complexity

• learning of optimal pairs for ORB/FREAK

• Is evaluation of paper exhaustive?

• real time?

• object tracking?



Learning Algorithm (ORB)






Learning Algorithm (FREAK)


Two Supplementary Evaluation PapersBekele, D., Teutsch, M., & Schuchert, T. (2013). Evalua-tion of binary keypoint descriptors. In Image Proces-sing (ICIP), 2013 20th IEEE International Conference (pp.3652-3656). doi: 10.1109/ICIP.2013.6738753

compares BRIEF, ORB, BRISK andFREAK

data sets: Oxford and Stanford MobileVisual Search (the last w/o groundtruth values)

conlusion: paper recommends BRISK

BRISK needs significant morecomputational effort compared to ORB

FREAK faster than BRISK, has lessmemory load, slightly less performance

SIFT has state-of-the-art performance

Figat, J., Kornuta, T., & Kasprzak, W. (2014). PerformanceEvaluation of Binary Descriptors of Local Features. InComputer Vision andGraphics (pp. 187-194). Springer In-ternational Publishing.

compares BRIEF, ORB, BRISK andFREAK; each with all kinds of detectors

data set: Oxford

conclusion: recommends ORB detectorwith BRISK descr. or FREAK descr.(depending on distortion type)

FREAK for viewpoint changes muchbetter than BRISK/ORB

SIFT has state-of-the-art performance


Two Supplementary Evaluation Papers

Bekele, D., Teutsch, M., & Schuchert, T. (2013). Evaluation of binary keypoint descriptors. In ImageProcessing (ICIP), 2013 20th IEEE International Conference (pp. 3652-3656). doi:10.1109/ICIP.2013.6738753


Two Supplementary Evaluation Papers

Bekele, D., Teutsch, M., & Schuchert, T. (2013). Evaluation of binary keypoint descriptors. In ImageProcessing (ICIP), 2013 20th IEEE International Conference (pp. 3652-3656). doi:10.1109/ICIP.2013.6738753


Date post:	03-Dec-2018
Category:	Documents
Upload:	dangtuyen
View:	214 times
Download:	0 times

An Update on Features - SURF, BRISK, ORB and FREAK · SURF boxfilterpyramid Haarwavelets binary...

Documents