CVPR 2013 Diversity Tutorial
Closing Remarks: What can we do with multiple
diverse solutions?
Dhruv Batra Virginia Tech
CVPR 2013 Diversity Tutorial
(C) Dhruv Batra 2
CRF
. . .
Diverse Segmentations
Example Result
Now what?
CVPR 2013 Diversity Tutorial
Your Options • Nothing
– User in the loop
• (Approximate) Min Bayes Risk – Use solutions to estimate the distribution and optimize Bayes
Risk
• Re-ranking – Pick a good solution from the list
(C) Dhruv Batra 3
Increasing Side Information
CVPR 2013 Diversity Tutorial
Interactive Segmentation • Setup
– Model: Color/Texture + Potts Grid CRF – Inference: Graph-cuts – Dataset: 50 train/val/test images
(C) Dhruv Batra 4
Image + Scribbles Diverse 2nd Best 2nd Best MAP MAP
1-2 Nodes Flipped 100-500 Nodes Flipped
CVPR 2013 Diversity Tutorial
Interactive Segmentation
(C) Dhruv Batra 5
89%
90%
91%
92%
93%
94%
95%
96%
MAP M-Best-MAP Confidence DivMBest
+0.05%
+1.61%
+3.62%
(Oracle) (Oracle) (Oracle)
M=6
Seg
men
tatio
n A
ccur
acy
Better
CVPR 2013 Diversity Tutorial
Your Options • Nothing
– User in the loop
• (Approximate) Min Bayes Risk – Use solutions to estimate the distribution and optimize Bayes
Risk
• Re-ranking – Pick a good solution from the list
(C) Dhruv Batra 6
CVPR 2013 Diversity Tutorial
Statistics 101 • Loss
– PCP, Pascal Loss, etc
• “True” Distribution
• Expected Loss:
• Min Bayes Risk
(C) Dhruv Batra 7
XMAP
X
P (X )P (ygt | x) =
L(ygt, y)
BR(y) = EP
�L(ygt, y)
�
�
ygt∈Y
L(ygt, y)P (ygt)miny∈Y
CVPR 2013 Diversity Tutorial
Structured Output Problems • Min Bayes Risk
• Two Problems
• Approximate MBR:
(C) Dhruv Batra 8
Intractable Intractable
XMAP
X
P (X )
�
ygt∈Y
L(ygt, y)P (ygt)miny∈Y
miny∈DivMBest
�
ygt∈DivMBest
L(ygt, y)P (ygt)
CVPR 2013 Diversity Tutorial
Semantic Segmentation • Setup
– Models: • Hierarchical CRF [Ladicky et al. ECCV ’10, ICCV ‘09]
• Second-Order Pooling [Carreira ECCV ‘12]
– Inference: • Alpha-expansion • Greedy
– Dataset: Pascal Segmentation Challenge (VOC 2012) • 20 categories + background; ~1500 train/val/test images
(C) Dhruv Batra 9
CVPR 2013 Diversity Tutorial
(C) Dhruv Batra 10
CRFDiverse Segmentations
Large-Margin Re-ranking
CVPR 2013 Diversity Tutorial
Semantic Segmentation
(C) Dhruv Batra 11
Input MAP Best of 10-Div
CVPR 2013 Diversity Tutorial
Semantic Segmentation
(C) Dhruv Batra 12
PAC
AL
Acc
urac
y
Better
#Solutions / Image
44%
47%
50%
53%
56%
59%
1 2 3 4 5 6 7 8 9 10
MAP [State-of-art circa 2012]
15%-gain possible
Same Features Same Model
DivMBest (Oracle)
Rand (Re-rank)
MBR
CVPR 2013 Diversity Tutorial
Your Options • Nothing
– User in the loop
• (Approximate) Min Bayes Risk – Use solutions to estimate the distribution and optimize Bayes
Risk
• Re-ranking – Pick a good solution from the list
(C) Dhruv Batra 13
CVPR 2013 Diversity Tutorial
(C) Dhruv Batra 14
CRFDiverse Segmentations
Large-Margin Re-ranking
CVPR 2013 Diversity Tutorial
(C) Dhruv Batra 15
CRFDiverse Segmentations
Large-Margin Re-ranking
ψ( , )
CVPR 2013 Diversity Tutorial
(C) Dhruv Batra 16
CRFDiverse Segmentations
Large-Margin Re-ranking
ψ( , )α� −α�ψ( , )
CVPR 2013 Diversity Tutorial
(C) Dhruv Batra 17
CRFDiverse Segmentations
Large-Margin Re-ranking
ψ( , )α� −α�ψ( , ) ≥ 1− ξilossi
minα,ξi
||α||2 + C�
i
ξiDiscriminative Re-ranking of Diverse Segmentation
[Yadollahpour et al., CVPR13, Wednesday Poster]
CVPR 2013 Diversity Tutorial
Semantic Segmentation
(C) Dhruv Batra 18
PAC
AL
Acc
urac
y
Better
#Solutions / Image
44%
47%
50%
53%
56%
59%
1 2 3 4 5 6 7 8 9 10
MAP [State-of-art circa 2012]
DivMBest (Oracle)
Rand (Re-rank)
DivMBest (Re-ranked) [Y.B.S., CVPR ‘13]
MBR
CVPR 2013 Diversity Tutorial
Qualitative Results: Success
(C) Dhruv Batra 19
CVPR 2013 Diversity Tutorial
Qualitative Results: Success
(C) Dhruv Batra 20
CVPR 2013 Diversity Tutorial
Qualitative Results: Success
(C) Dhruv Batra 21
CVPR 2013 Diversity Tutorial
Qualitative Results: Failures
(C) Dhruv Batra 22
CVPR 2013 Diversity Tutorial
Qualitative Results: Failures
(C) Dhruv Batra 23
CVPR 2013 Diversity Tutorial
Qualitative Results: Failures
(C) Dhruv Batra 24
CVPR 2013 Diversity Tutorial
Summary • All models are wrong
• Some beliefs are useful
• Diverse Multiple Solutions – A way to get useful beliefs out.
• DivMBest + Reranking – Big impact possible on many applications!
(C) Dhruv Batra 25
CVPR 2013 Diversity Tutorial
Summary
• What does my model believe?
(C) Dhruv Batra 26
�
Posterior Summary
CVPR 2013 Diversity Tutorial
Thanks!
(C) Dhruv Batra 27