3D Model Assisted Image Segmentation
Srimal Jayawardena Di Yang Marcus HutterAustralian National University
srimal(dot)jayawardena(at)anu(dot)edu(dot)auhttp://users.cecs.anu.edu.au/˜srimalj
DICTA 2011
Srimal Jayawardena Australian National University 3D Model Assted. Img. Seg. DICTA 2011 1 / 15
The problem
Segmenting a mostly homogeneous (same color/texture) object intoparts is a hard problem.
(a) Original Image (b) Segmentated into parts
Srimal Jayawardena Australian National University 3D Model Assted. Img. Seg. DICTA 2011 2 / 15
Methodology Overview
Pose estimation(hierarchically minimise gradient based loss)
Contour detection
Segmented parts
Photo of known object
(Input)
3D CAD Model (Input)
Project 3D model parts (initialise contour detection)
Srimal Jayawardena Australian National University 3D Model Assted. Img. Seg. DICTA 2011 3 / 15
Gradient Loss for Pose EstimationLet θ parameterize the pose of the 3D model w.r.t the camera.
(a) 3D Model Gradients GN(θ) (b) Photo Gradients GI
Loss at pose θ,
Lg(θ) := 1− (corr( GN(θ) , GI ))2 ∈ [0, 1]
Srimal Jayawardena Australian National University 3D Model Assted. Img. Seg. DICTA 2011 4 / 15
3D Model Gradients
(a) Φx(u, v,θ) (b) ∂Φx(u,v,θ)∂u
(c) ∂Φx(u,v,θ)∂v
(d) Φy(u, v,θ) (e) ∂Φy(u,v,θ)
∂u(f) ∂Φy(u,v,θ)
∂v
(g) Φz(u, v,θ) (h) ∂Φz(u,v,θ)∂u
(i) ∂Φz(u,v,θ)∂v
(j) GN(θ)
GN (θ)(u, v) = ||∇Φ(u, v,θ)||kk (1)
Srimal Jayawardena Australian National University 3D Model Assted. Img. Seg. DICTA 2011 5 / 15
Photo Gradients
(a) Real photo (b) SyntheticUsing grayscale intensity
(c) Real ∂I∂u
(d) Synthetic ∂I∂u
(e) Real ∂I∂v
(f) Synthetic ∂I∂v
)
(g) Real GI (h) Synthetic GI
GI(u, v) = ||∇I(u, v)||kk (2)
Srimal Jayawardena Australian National University 3D Model Assted. Img. Seg. DICTA 2011 6 / 15
Overlays and Smoothing
(a) Real (b) n=0 (c) n=2
(d) Synthetic (e) n=0 (f) n=2
Srimal Jayawardena Australian National University 3D Model Assted. Img. Seg. DICTA 2011 7 / 15
Loss Landscapes
0.975
0.98
0.985
0.99
0.995
1
-15 -10 -5 0 5 10 15
Loss
Valu
e
Percentage shift in x direction from known pose
n = 0n = 1n = 2
(a) 2-norm
0.75
0.8
0.85
0.9
0.95
1
-15 -10 -5 0 5 10 15
Loss
Valu
ePercentage shift in x direction from known pose
n = 0n = 1n = 2
(b) 1-norm
Srimal Jayawardena Australian National University 3D Model Assted. Img. Seg. DICTA 2011 8 / 15
Hierarchical Optimization
(a) Photo (b) Background removed
(c) n=2 (d) n=1 (e) n=0
(f) Final fine pose n=0
Next: Initialise a Level Set Evolution contour detection from projected 3Dmodel parts
Srimal Jayawardena Australian National University 3D Model Assted. Img. Seg. DICTA 2011 9 / 15
Contour DetectionLevel Set Evolution without re-initialization [Li et al., 2005, CVPR]
Row 1: Level set function, Row 2: Zero level curve
(a) Initialisation (b) (c) (d) Final
Srimal Jayawardena Australian National University 3D Model Assted. Img. Seg. DICTA 2011 10 / 15
Results
(a) Initialisation (b) Result (c) Benchmark GC (d) Benchmark LS
(e) Initialisation (f) Result (g) Benchmark GC (h) Benchmark LS
Srimal Jayawardena Australian National University 3D Model Assted. Img. Seg. DICTA 2011 11 / 15
Results
(a) Initialisation (b) Result (c) Benchmark GC (d) Benchmark LS
(e) Initialisation (f) Result (g) Benchmark GC (h) Benchmark LS
Srimal Jayawardena Australian National University 3D Model Assted. Img. Seg. DICTA 2011 12 / 15
Accuracy
Part segmentation results for two views of a Mazda Astina.
Accuracy = 1−(
No.Misclassified.P ixelsNo.Ground.Truth.P ixels
)Part Side View Semi Profile Avg.Fender 97.7% 97.6% 97.7%Front door 98.1% 95.3% 96.7%Back door 96.8% 93.6% 95.2%Mud flap 97.3% 95.1% 96.2%Front window 97.8% 97.5% 97.7%Back window 99.5% 93.9% 96.7%
Srimal Jayawardena Australian National University 3D Model Assted. Img. Seg. DICTA 2011 13 / 15
Discussion
Challenges - High amount of reflections and noiseA closer initialisation curve - better resultsFuture work - simlutaneous pose estimation and segmentation
Thank you!
Srimal Jayawardena Australian National University 3D Model Assted. Img. Seg. DICTA 2011 14 / 15
References I
Li, C., Xu, C., Gui, C., and Fox, M. (2005).
Level set evolution without re-initialization: a new variationalformulation.
In Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEEComputer Society Conference on, volume 1, pages 430 – 436 vol. 1.
Srimal Jayawardena Australian National University 3D Model Assted. Img. Seg. DICTA 2011 15 / 15
