Post on 20-Mar-2017
transcript
Energy Minimization with Label Costsand Model Fitting
presented by Yuri Boykov
co-authors:Andrew Delong Anton Osokin Hossam Isack
The University of
OntarioOverview Standard models in vision (focus on discrete case)
• MRF/CRF, weak-membrane, discontinuity-preserving...• Information-based: MDL (Zhu&Yuille’96) , AIC/BIC (Li’07)
Label costs and their optimization • LP-relaxations, heuristics, α-expansion++
Model Fitting• dealing with infinite number of labels ( PEARL )
Applications• unsupervised image segmentation• geometric model fitting (lines, circles, planes, homographies, ...)• rigid motion estimation• extensions…
The University of
Ontario
Reconstruction in Vision: (a basic example)
L
observed noisy image I image labeling L(restored intensities)
How to compute L from I ?
I
L = { L1, L2 , ... , Ln }I = { I1, I2 , ... , In }
The University of
Ontario
Energy minimization(discrete approach)
MRF framework• weak membrane model (Geman&Geman’84, Blake&Zisserman’83,87)
pL qL
ZZLp 2:
Nqp
qpp
pp LLVILE),(
2 ),()()(L ,V
T Tdiscontinuity preserving potentials
Blake&Zisserman’83,87
spatial regularizationdata fidelity
The University of
OntarioOptimization Convex regularization
• gradient descent works• exact polynomial algorithms
TV regularization• a bit harder (non-differentiable)• global minima algorithms (Ishikawa, Hochbaum, Nikolova et al.)
Robust regularization• NP-hard, many local minima• good approximations (message passing, a-expansion)
,V
,V
,V
Nqp
qpp
pp LLVILE),(
2 ),()()(L
The University of
Ontario
Potts model(piece-wise constant labeling)
Robust regularization• NP-hard, many local minima• provably good approximations (a-expansion)
,V
)(),( TwV
maxflow/mincut combinatorial algorithms
Nqp
qpp
pp LLVILE),(
2 ),()()(L
The University of
Ontario
Left eye imageRight eye image
Potts model(piece-wise constant labeling)
Robust regularization• NP-hard, many local minima• provably good approximations (a-expansion)
,V
)(),( TwV
depth layers
maxflow/mincut combinatorial algorithms
Nqp
qpp
pp LLVLDE),(
),()()(L
The University of
Ontario
Potts model(piece-wise constant labeling)
Robust regularization• NP-hard, many local minima• provably good approximations (a-expansion)
,V
)(),( TwV
0
C
1
maxflow/mincut combinatorial algorithms
Nqp
qpp
pp LLVLDE),(
),()()(L
The University of
OntarioAdding label costs
Lippert [PAMI 89]• MDL framework, annealing
Zhu and Yuille [PAMI 96]• continuous formulation (gradient des cent )
H. Li [CVPR 2007]• AIC/BIC framework, only 1st and 3rd terms• LP relaxation (no guarantees approximation)
Our new work [CVPR 2010] , extended a-expansion • all 3 terms, 3rd term is represented as some high-order clique), optimality bound• very fast heuristics for 1st & 3rd term (facility location problem, 60-es)
L
LLN)q,p(
qpp
pp )(h)L,L(V)L(D)(E LL
- set of labelsallowed at each
point p
otherwise
LLp pL ,0
:,1)(L
The University of
OntarioThe rest of the talk…
Why label costs?
The University of
OntarioModel fitting
pL
||Lp||minargL̂
}b,a{L
p
2xy )bapp(||Lp||
y=ax+b
SSD
The University of
Ontariomany outliersquadratic errors fail
use more robust error measures, e.g.
gives “MEDIAN” line|bapp|||Lp|| xy
- more expensive computations
(non-differentiable)- still fails if
outliers exceed 50%
RANSAC
The University of
Ontariomany outliers
RANSAC
1. sample randomlytwo points, get a line
The University of
Ontariomany outliers
10 inliers
RANSAC
1. sample randomlytwo points, get a line2. count inliers for
threshold T
The University of
Ontariomany outliers
30 inliers
1. sample randomlytwo points, get a line2. count inliers for
threshold T
3. repeat N times and select
model with most inliers
RANSAC
The University of
OntarioMultiple models and many outliers
Why not RANSAC
again?
The University of
OntarioMultiple models and many outliers
In general, maximization of inliersdoes not work for
outliers + multiple models
Why not RANSAC
again?
Higher noise
The University of
OntarioEnergy-based approach
p
||Lp||)L(E
energy-based interpretation of RANSAC criteria forsingle model fitting:
- find optimal label Lfor one very specific
error measure
TdistifTdistif
dist,1,0
||||
The University of
OntarioEnergy-based approach
Npq
qpp
p LLTwLpE )(||||)(L
If multiple models
- assign different models (labels Lp) to every point
p
- find optimal labelingL = { L1, L2 , ... , Ln }
Need regularization!
The University of
OntarioEnergy-based approach
Npq
qpp
p LLTwLpE )(||||)(L
If multiple models
- assign different models (labels Lp) to every point
p
- find optimal labelingL = { L1, L2 , ... , Ln }
The University of
OntarioEnergy-based approach
If multiple models
- assign different models (labels Lp) to every point
p
- find optimal labelingL = { L1, L2 , ... , Ln }
L
LLp
p )(h||Lp||)(E LL
otherwise
LLp pL ,0
:,1)(L
- set of labelsallowed at each
point p
The University of
OntarioEnergy-based approach
If multiple models
- assign different models (labels Lp) to every point
p
- find optimal labelingL = { L1, L2 , ... , Ln }
L
LLN)q,p(
qpp
p )(h)LL(Tw||Lp||)(E LL
Practical problem: number of potential labels (models) is huge, how are we going to use a-expansion?
The University of
OntarioPEARL
data points
ProposeExpandAndReestimateLabels
The University of
OntarioPEARL
data points + randomly sampled models
sample datato generatea finite set
of initial labels
ProposeExpandAndReestimateLabels
The University of
OntarioPEARL
models and inliers (labeling L)
a-expansion:minimize E(L)
segmentationfor fixed
set of labels
Npq
qpp
p )LL(Tw||Lp||)(E L
ProposeExpandAndReestimateLabels
The University of
OntarioPEARL
ProposeExpandAndReestimateLabels
models and inliers (labeling L)
reestimating labels in
for given inliers
minimizes first term
of energy E(L)
Npq
qpp
p )LL(Tw||Lp||)(E L
The University of
OntarioPEARL
ProposeExpandAndReestimateLabels
models and inliers (labeling L)
Npq
qpp
p )LL(Tw||Lp||)(E L
a-expansion:minimize E(L)
segmentationfor fixed
set of labels
The University of
OntarioPEARL
iterate until convergence
Npq
qpp
p )LL(Tw||Lp||)(E L
after 5 iterations
ProposeExpandAndReestimateLabels
The University of
Ontario
PEARL can significantlyimprove initial models
single line fittingwith 80% outliers
number of initial samplesde
viat
ion
(fr
om g
roun
d tr
uth
)
The University of
Ontario
Comparison formulti-model fitting
original data points
Low noise
The University of
Ontario
Comparison formulti-model fitting
some generalization of RANSAC
Low noise
The University of
Ontario
Comparison formulti-model fitting
PEARL
Low noise
The University of
Ontario
Comparison formulti-model fitting
original data points
High noise
The University of
Ontario
Comparison formulti-model fitting
Some generalization of RANSAC (Multi-RANSAC, Zuliani et al. ICIP’05)
High noise
The University of
Ontario
Comparison formulti-model fitting
Other generalization of RANSAC (J-linkage, Toldo & Fusiello, ECCV’08)
High noise
The University of
Ontario
Comparison formulti-model fitting
Finding modes in Hough-space, e.g. via mean-shift(also maximizes the number of inliers)
Hough transform
High noise
The University of
Ontario
Comparison formulti-model fitting
PEARL
High noise
The University of
OntarioFitting circles
Here spatial regularization does not work well
regularization with label costs only
The University of
OntarioFitting planes (homographies)
Original image (one of 2 views)
The University of
OntarioFitting planes (homographies)
(a) Label costs only
The University of
OntarioFitting planes (homographies)
(b) Spatial regularity only
The University of
OntarioFitting planes (homographies)
(c) Spatial regularity + label costs
The University of
Ontario
(unsupervised) Image Segmentation
Original image
The University of
Ontario
(unsupervised) Image Segmentation
(a) Label costs only [Li, CVPR 2007]
The University of
Ontario
(unsupervised) Image Segmentation
(b) Spatial regularity only [Zabih&Kolmogorov CVPR 04]
The University of
Ontario
(unsupervised) Image Segmentation
(c) Spatial regularity + label costs
Zhu and Yuille 96used continuous
variational formulation(gradient discent)
The University of
Ontario
(unsupervised) Image Segmentation
(c) Spatial regularity + label costs
The University of
Ontario
(unsupervised) Image Segmentation
Spatial regularity + label costs
The University of
Ontario
(unsupervised) Image Segmentation
Spatial regularity + label costs
The University of
Ontario
(unsupervised) Image Segmentation
Spatial regularity + label costs
The University of
Ontario
(unsupervised) Image Segmentation
Spatial regularity + label costs
The University of
Ontario
(rigid)Motion Estimation
Original image
3 motions
[Rene Vidal]
The University of
Ontario
(rigid)Motion Estimation
(a) Label costs only
3 motions
The University of
Ontario
(rigid)Motion Estimation
(b) Spatial regularity only
7 motions
The University of
Ontario
(rigid)Motion Estimation
(c) Spatial regularity + label costs
3 motions
The University of
OntarioPlane fitting
Note very small steps between each floor
The University of
Ontario
Affine model fitting(from a rectified stereo pair)
photoconsistency + smoothness
dense model assignments to pixels
Birchfield & Tomasi’99(fit initial models to output of other stereo
algorithm ++ α-expansion + reestimation)
geometric errors + smoothness+ label cost
sparse model assignments to features
PEARL(sample data + α-expansion + reestimation)
The University of
Ontario
Duh...use right geometric error measure!!!
“disparity” errors d1 and d2 (bad idea!)“quatient”-based errors d (standard)
The University of
Ontario
Affine model fitting(from a rectified stereo pair)
photoconsistency + smoothness
dense model assignments to pixels
Birchfield & Tomasi’99(fit initial models to output of other stereo
algorithm ++ α-expansion + reestimation)
geometric errors + smoothness+ label cost
sparse model assignments to features
PEARL(sample data + α-expansion + reestimation)
The University of
Ontario
Photoconsistency vs. Geometric Alignment
photoconsistency optimization (Birchfield & Tomasi’99)
densestereo
The University of
Ontario
Photoconsistency vs. Geometric Alignment
geometric error minimization via PEARL
sparse data
sparsestereo