Unsupervised Detection of Regions of Interest
Using Iterative Link Analysis
Gunhee Kim1 Antonio Torralba2
1: SCS, CMU2: CSAIL, MIT
Neural Information Processing Systems 2009
November 30, 2009
Unsupervised Detection of ROIs
A set of images…
Rectangular Regions of Interest
Why Is the ROI Detection Useful ?
Scene recognition[Quattoni&Torralba, CVPR09]
Training for Recognition[Bosch et al, ICCV07]
Flickr Notes
Alternating Optimization
• One of widely used heuristics for iterative optimization
• Optimization over two sets of variables is not easy
• But affordable to optimize one while the other is fixed
• Goal: Find correspondences between two sets of point clouds [Besl&McKay,1992]
Example – Iterative Closest Point Algorithm
Trans-formation
Estimate transformation parameters
Corres-pondences
Associate points by NN criteria
• Goal: Clustering
Example – K-means
ClusterMembershi
p
Find nearest cluster center
ClusterCenters
Take mean
Initialization
Pictures from Bishop’s book
• Goal: Find best ROIs in each image of dataset
Unsupervised Detection of ROIs
RefineROIs
Detection or Localization
FindExamplars
Modeling or Ranking
examplars
Where is butterfly?
What are examplars?
Our Approach
• Inspired by alternating optimization
• Based on link analysis of hypothesis network.
• Find Examplars = Central and diverse Hubs
• Refine ROIs = Highly-ranked Hypotheses in each image wrt examplars
• Easy, Fast and Dynamic– Simple heuristic for linearity of computation wrt dataset size.– Ex. 4.5 hours / 200k images with naïve matlab implementation.
ROI Candidates and Description
• For each , define– At least one of would be good
• Description: Spatial pyramids of visual words and HOG
• Similarity measure: Cosine similarity
An image 15 segments 43 ROI hypotheses
Visual words Edge Gradient
Algorithm - Input
• Image set and its ROI hypothesis set
Algorithm - Initialization
• Best ROI = Image itself !
Algorithm - Initialization
• Initialization is essential for the success !
• Why is it a feasible idea for Web images ?– Most pictures are taken from a canonical view so that an object
of interest is located in a center with significant size.– Given a similarity network of a sufficiently large number of
images, democratic voting reveals the most dominant visual information as hubs [Kim et al 08]
Examples of top-ranked Images
Algorithm – First Hub Seeking
• Generate a similarity network and find a hub set
Algorithm – First ROI Refinement
• Bipartite graph between hub sets and All ROIs of an image
Algorithm – Second Hub Seeking
• Keep iterating…
Hub Seeking with Centrality & Diversity
• Mean-shift like hub seeking algorithm
Mean Shift[Comaniciu and Meer,
PAMI 2002]
K-NN similarity matrix PageRank vector
G(t)
K-NN graph
Degree distribution ~ PageRank vector
Hub Seeking with Centrality & Diversity
• Mean-shift like hub seeking algorithm
0.05
0.20.5
0.25
0.80.5
0.1
Max P-value !
Fixed radius window= max. reachable probability d (= 0.1)
Mean Shift
ROI Refinement
• Augmented Bipartite Graph
(1-α)Wo
WoT
αWi
ROI hypothesis Hub set vector
RO
I hyp
othe
ses
Hub
set
PageRank
Argmax ( )i
ROI Refinement
• What does α do?
(1-α)Wo
WoT
αWi
α = 0 α = 0.1
Wo
WoT
Example - ROI Refinement
T=0 T=1 T=2 T=3 T=4 T=5 T=6 T=7
T=0 T=1 T=2 T=3 T=4 T=5 T=6 T=7
Scalability Setting
• Bottleneck: Quadratic computation to generate a similarity matrix of selected ROIs
• If dataset size is too large, – Run the algorithm with N number of images (N = 10,000)– Re-use x % of previous high-ranked images.
Dataset
N
N N
N
Experiments
• Performance Test– PASCAL VOC 2006 Dataset– Weakly-supervised1 and Unsupervised2
• Scalability Test– Five objects: {butterfly+insect (69,990), classic+car (265,731),
motorcycle+bike (106,590), sunflower (165,235), giraffe+zoo (53,620)}
– Weakly-supervised1
1: Input imageset consists of a single object type (only localization is required) 2: Input imageset consists of multiple object types (localization and clustering are required)
Performance Tests
• Weakly Supervised Localization (PR-Curves)
[Russell et al. CVPR 2006]http://www.di.ens.fr/russell/projects/mult seg discovery/index.html
X-axis: RecallY-axis: Precision
Performance Tests
• Unsupervised Classification & Localization
X-axis: RecallY-axis: Precision
X-axis: FP rateY-axis: TP rate
ROCCurves
PRCurves
Scalability Tests
• Weakly-supervised Localization
X-axis: RecallY-axis: Precision
Perturbation Tests
• Robustness of ROI detection of each image against random network formation – 100 random sets of size of 200 images
Entropy: 0.2419 1.6846 2.4331
Dataset
An image of interest
X-axis: ROI hypothesesY-axis: Frequencies
Localization Examples
Conclusion
• Alternating optimization based Unsupervised ROI detection
• Simple and Fast
• Competitive performance on PASCAL 06
• Scalable Test with more than 200K Flickr images
• Critic: Analysis for convexity, convergence, sensitivity to initialization, quality of solution
Algorithm