Object retrieval with large vocabularies and fast spatial
matching
James Phibin1, Ondrej Chum1, Michael Isard2,Josef Sivic1, and Andrew Zisserman1
1Department of Engineering Science, 2University of OxfordMicrosoft Research,Silicon Valley
CVPR 2007
Overview
• Problem– Input: a user-selected region of a query image– Return: a ranked list of images retrieved from a large
corpus.• Containing the same object
• Objective– a promising step towards “web-scale” image corpora
• Improvement– Improving the visual vocabulary– Incorporating spatial information into the ranking– Examples
Datasets
• Source– Flickr
• Oxford 5K dataset– “Oxford Christ Church,” “Oxford
Radcliffe Camera,”… with “Oxford”
– 5,062 (1,024*768) images
• 100K dataset– 145 most popular tags– 99,782 (1,024*768) images
• 1M dataset– 450 most popular tags– 1,040,801 (500*333) images
Indexing the dataset
• Image description– Affine-invariant Hessian regions
• 3,300 regions on a 1,024*768 image– SIFT descriptor
• 128-D– 4×4× 8-direction gradient histogram
• Model– bag-of-visual-words
• Quantize the visual descriptors to index the image
• Search engine– L2 distance as similarity– tf-idf weighting scheme
• more commonly occurring = less discriminative = smaller weight
2×2 8-direction gradient histogram
Train the Dictionary
K-mean
Approximate k-mean (AKM)
Hierarchical k-mean (HKM)
• Traditional k-mean– single iteration
• O(NK)
• Strategy– Reduce the number of candidates of nearest cluster heads– AKM
• Approximate nearest neighbor– replace the exact computing nearest neighbors with
» 8 randomized k-d tree of cluster heads• Less than 1% of points are assigned differently from k-mean for moderate values of K
– HKM• “vocabulary tree”
– A small number (K=10) of cluster centers at each level– Kn clusters at the n-th level
• Quantization effect– AKM
• Conjunction of trees– Overlapping partition
– HKM• Points can additionally be assigned to some internal nodes
AKM v.s.HKM2D k-d tree
Comparing vocabularies
K-mean v.s. AKM
HKM v.s.AKM
Scaling up with AKM
Ground Truth
• Dataset– 5K dataset
• Searching– Manual– Entire– For 11 landmarks
• Labels– Positive
• Good: nice, clear• OK: more than 25% of the object
– Null• Junk: less than 25%
– Negative• Absent: object not present
5 queries for each landmark
Evaluation
• Precision– # of retrieved positive images / # of total retrieved
images
• Recall– # of retrieved positive images / # of total positive
images
• Average precision (AP)– The area under the precision-recall curve for a query
• Mean average precision (mAP)• Average AP for each of the 5 queries for a landmark• Final mAP = average for mAP for each landmark
K-mean v.s. AKM
HKM v.s.AKM
Recognition Benchmark
D. Nistér and H. Stewénius. Scalable recognition with a vocabulary tree. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), volume 2, pages 2161-2168, June 2006.
Scaling up with AKM
Spatial re-ranking
Use Spatial Info.
• Usage– Re-ranking the top ranked results
• Procedure1. Estimate a transformation for each target image2. Refine the estimations
– Reduce the errors due to outliers– LO-RANSAC
» RANdom SAmple Consensus » Additional modeL Optimization step
3. Re-rank target images– Scoring target images to the sum of the idf value for the
inlier words– Verified images above unverified images
Restricted transformation
• Degree of freedom– 3 dof
• Isotropic scale• Covering the changes in zoom or
distance– 4 dof
• Anisotropic scale• Covering foreshortening, either horizontal
or vertical– 5 dof
• Anisotropic scale and vertical shear
• NOT– In-plane rotation
foreshorten(perspective)
shear
Comparing spatial rankings
Different transformation typesLarge datasets
ExamplesExamples of errors
Different transformation types
Large datasets
Examples
Examples of errors
Conclusion
• Conclusion– Scalable visual object-retrieval system
• Future work– More evaluation for higher scale– Including spatial info. into the index– Moving some of the burden of spatial
matching to the first ranking stage
RANSAC
http://en.wikipedia.org/wiki/RANSAC
RANSAC example
