Unsupervised Detection of Regions of Interest

Using Iterative Link Analysis

Gunhee Kim1 Antonio Torralba2

1: SCS, CMU2: CSAIL, MIT

Neural Information Processing Systems 2009

November 30, 2009

Unsupervised Detection of ROIs

A set of images…

Rectangular Regions of Interest

Why Is the ROI Detection Useful ?

Scene recognition[Quattoni&Torralba, CVPR09]

Training for Recognition[Bosch et al, ICCV07]

Flickr Notes

Alternating Optimization

• One of widely used heuristics for iterative optimization

• Optimization over two sets of variables is not easy

• But affordable to optimize one while the other is fixed

• Goal: Find correspondences between two sets of point clouds [Besl&McKay,1992]

Example – Iterative Closest Point Algorithm

Trans-formation

Estimate transformation parameters

Corres-pondences

Associate points by NN criteria

• Goal: Clustering

Example – K-means

ClusterMembershi

p

Find nearest cluster center

ClusterCenters

Take mean

Initialization

Pictures from Bishop’s book

• Goal: Find best ROIs in each image of dataset

Unsupervised Detection of ROIs

RefineROIs

Detection or Localization

FindExamplars

Modeling or Ranking

examplars

Where is butterfly?

What are examplars?

Our Approach

• Inspired by alternating optimization

• Based on link analysis of hypothesis network.

• Find Examplars = Central and diverse Hubs

• Refine ROIs = Highly-ranked Hypotheses in each image wrt examplars

• Easy, Fast and Dynamic– Simple heuristic for linearity of computation wrt dataset size.– Ex. 4.5 hours / 200k images with naïve matlab implementation.

ROI Candidates and Description

• For each , define– At least one of would be good

• Description: Spatial pyramids of visual words and HOG

• Similarity measure: Cosine similarity

An image 15 segments 43 ROI hypotheses

Visual words Edge Gradient

Algorithm - Input

• Image set and its ROI hypothesis set

Algorithm - Initialization

• Best ROI = Image itself !

Algorithm - Initialization

• Initialization is essential for the success !

• Why is it a feasible idea for Web images ?– Most pictures are taken from a canonical view so that an object

of interest is located in a center with significant size.– Given a similarity network of a sufficiently large number of

images, democratic voting reveals the most dominant visual information as hubs [Kim et al 08]

Examples of top-ranked Images

Algorithm – First Hub Seeking

• Generate a similarity network and find a hub set

Algorithm – First ROI Refinement

• Bipartite graph between hub sets and All ROIs of an image

Algorithm – Second Hub Seeking

• Keep iterating…

Hub Seeking with Centrality & Diversity

• Mean-shift like hub seeking algorithm

Mean Shift[Comaniciu and Meer,

PAMI 2002]

K-NN similarity matrix PageRank vector

G(t)

K-NN graph

Degree distribution ~ PageRank vector

Hub Seeking with Centrality & Diversity

• Mean-shift like hub seeking algorithm

0.05

0.20.5

0.25

0.80.5

0.1

Max P-value !

Fixed radius window= max. reachable probability d (= 0.1)

Mean Shift

ROI Refinement

• Augmented Bipartite Graph

(1-α)Wo

WoT

αWi

ROI hypothesis Hub set vector

RO

I hyp

othe

ses

Hub

set

PageRank

Argmax ( )i

ROI Refinement

• What does α do?

(1-α)Wo

WoT

αWi

α = 0 α = 0.1

Wo

WoT

Example - ROI Refinement

T=0 T=1 T=2 T=3 T=4 T=5 T=6 T=7

T=0 T=1 T=2 T=3 T=4 T=5 T=6 T=7

Scalability Setting

• Bottleneck: Quadratic computation to generate a similarity matrix of selected ROIs

• If dataset size is too large, – Run the algorithm with N number of images (N = 10,000)– Re-use x % of previous high-ranked images.

Dataset

N

N N

N

Experiments

• Performance Test– PASCAL VOC 2006 Dataset– Weakly-supervised1 and Unsupervised2

• Scalability Test– Five objects: {butterfly+insect (69,990), classic+car (265,731),

motorcycle+bike (106,590), sunflower (165,235), giraffe+zoo (53,620)}

– Weakly-supervised1

1: Input imageset consists of a single object type (only localization is required) 2: Input imageset consists of multiple object types (localization and clustering are required)

Performance Tests

• Weakly Supervised Localization (PR-Curves)

[Russell et al. CVPR 2006]http://www.di.ens.fr/russell/projects/mult seg discovery/index.html

X-axis: RecallY-axis: Precision

Performance Tests

• Unsupervised Classification & Localization

X-axis: RecallY-axis: Precision

X-axis: FP rateY-axis: TP rate

ROCCurves

PRCurves

Scalability Tests

• Weakly-supervised Localization

X-axis: RecallY-axis: Precision

Perturbation Tests

• Robustness of ROI detection of each image against random network formation – 100 random sets of size of 200 images

Entropy: 0.2419 1.6846 2.4331

Dataset

An image of interest

X-axis: ROI hypothesesY-axis: Frequencies

Localization Examples

Conclusion

• Alternating optimization based Unsupervised ROI detection

• Simple and Fast

• Competitive performance on PASCAL 06

• Scalable Test with more than 200K Flickr images

• Critic: Analysis for convexity, convergence, sensitivity to initialization, quality of solution

Algorithm