SpeakerMin-Koo Kang
November 14, 2012
Depth Enhancement Technique by Sensor Fusion:Joint Bilateral Filter Approaches
22012-11-14 / Computer Vision Laboratory Seminar
Depth enhancement technique by sensor fusion
Outline1. Introduction
- Why depth data is important- How to acquire depth data- Depth upsampling: state-of-the-art approach
2. Background- Interpolation filters: Nearest Neighbor / Bilinear / Bicubic / Bilateral
3. Bilateral filter-based depth upsampling- Joint Bilateral Upsampling (JBU) filter / SIGGRAPH 2007- Pixel Weighted Average Strategy (PWAS) / ICIP 2010- Unified Multi-Lateral (UML) filter / AVSS 2011- Generalized depth enhancement framework / ECCV 2012
4. Concluding remarks
32012-11-14 / Computer Vision Laboratory Seminar
Depth enhancement technique by sensor fusion
- Why depth data is important- How to acquire depth data- State-of-the-art approaches
Introduction
42012-11-14 / Computer Vision Laboratory Seminar
Depth enhancement technique by sensor fusion
Why is depth data important? Used in various fields One of the most important techniques in
computer vision Important factors
speed, accuracy, resolution
3D reconstruction Virtual view generationIn 3DTV
Human computer interac-tion
52012-11-14 / Computer Vision Laboratory Seminar
Depth enhancement technique by sensor fusion
How to acquire depth data? Depth acquisition method comparison
Laser scanner Stereo vision Range sensor
Accuracy very high low(textureless, occlusion)
high
Speed slow Case by case real-time
Resolution high-resolution same as image low-resolution
Remarks only static scene
Laser scanning method Stereo vision sensor Range sensor
Can be overcome by depth map up-samplingRange sensor method has the most appropriate performance except low-resolution
62012-11-14 / Computer Vision Laboratory Seminar
Depth enhancement technique by sensor fusion
Problem definition Disparity estimation by range sensor delivers small
resolution of depth map Rendering requires full resolution depth map Main objectives / requirements:
- Cost-effective (potential for real-time at consumer electronics platforms)- Align depth map edge with image edge- Remove inaccuracies (caused by heuristics in disparity estimation)- Temporal stability (esp. at edges and areas with detail)
Upsam-pling Re-finement
72012-11-14 / Computer Vision Laboratory Seminar
Depth enhancement technique by sensor fusion
Depth upsampling Definition
Conversion of depth map with low resolution into one with high resolution
Approach Most state-of-the-art methods are based on sensor fusion
technique; i.e., use image sensor and range sensor together
Depth map up-samplingby using bi-cubic interpolation
Depth map up-sampling by using image and range sensor
82012-11-14 / Computer Vision Laboratory Seminar
Depth enhancement technique by sensor fusion
Background- Interpolation filters: Nearest Neighbor / Bilinear /
Bicubic / Bilateral
92012-11-14 / Computer Vision Laboratory Seminar
Depth enhancement technique by sensor fusion
Single Image-based Interpolation The conventional filterings
102012-11-14 / Computer Vision Laboratory Seminar
Depth enhancement technique by sensor fusion
The main types of artifacts are most easilyseen at sharp edges, and include aliasing (jagged edges), blurring, and edge halos (see il-lustration below)
Upsampling examples
0% Sharpening
16.7% Sharpening
25% Sharpening
Nearest Neighbor Bilinear BicubicInput
112012-11-14 / Computer Vision Laboratory Seminar
Depth enhancement technique by sensor fusion
Single Image-based Interpolation Bilateral filtering: smoothing an image without blurring its edges
122012-11-14 / Computer Vision Laboratory Seminar
Depth enhancement technique by sensor fusion
Bilateral filtering applicationsInput Gaussian
smoothingBilateral smoothing
noisy image
naïve denoisingby Gaussian filter
better denoisingby bilateral filter
132012-11-14 / Computer Vision Laboratory Seminar
Depth enhancement technique by sensor fusion
Bilateral filter-based depth upsampling- Joint Bilateral Upsampling (JBU) filter / SIGGRAPH 2007- Pixel Weighted Average Strategy (PWAS) / ICIP 2010- Unified Multi-Lateral (UML) filter / AVSS 2011- Generalized depth enhancement framework / ECCV 2012
142012-11-14 / Computer Vision Laboratory Seminar
Depth enhancement technique by sensor fusion
Joint bilateral filtering Multi-modal filtering
Range term defined by one modality Filtering performed on an other modality
Propagates properties from one to an other modality Edge preserving properties
152012-11-14 / Computer Vision Laboratory Seminar
Depth enhancement technique by sensor fusion
Joint bilateral upsampling (JBU) First publication on bilateral filters for upsampling at
SIGGRAPH 2007 J. Kopf, Univ. of Konstantz (Germany) provided reference sw.
[Kopf2007] solution: High resolution image in range term Low resolution input high resolution output
Kopf et al., “Joint Bilateral Upsampling”, SIGGRAPH 2007
162012-11-14 / Computer Vision Laboratory Seminar
Depth enhancement technique by sensor fusion
Joint bilateral upsampling (JBU) Representative formulation:
N(P): targeting pixel P(i, j)’s neighborhood. fS(.): spatial weighting term, applied for pixel position P. fI(.): range weighting term, applied for pixel value I(q). fS(.), fI(.) are Gaussian functions with standard deviations, σS and σI, respectively.
Kopf et al., “Joint Bilateral Upsampling”, SIGGRAPH 2007
Upsampleddepth map
Rendered3D view
172012-11-14 / Computer Vision Laboratory Seminar
Depth enhancement technique by sensor fusion
Is JBU ideal enough? Limitations of JBU:
It starts from the fundamental heuristic assumptions about the relationship between depth and intensity data
Sometimes depth has no corresponding edges in the 2-D image
Remaining problems: Erroneous copying of 2-D texture into actually smooth geometries within
the depth map Unwanted artifact known as edge blurring
High-resolution guidance image(red=non-visible depth discontinu-
ities)
Low-resolution depth map (red=zooming area)
JBU enhanced depth map(zoomed)
182012-11-14 / Computer Vision Laboratory Seminar
Depth enhancement technique by sensor fusion
Pixel Weighted Average Strategy (PWAS) Pixel Weighted Average Strategy for Depth Sensor Data
Fusion F. Garcia, proposed in ICIP 2010
[Garcia2010] solution: Use of a *credibility map to cope with texture copy & edge blurring Credibility map indicates unreliable regions in depth map
Representative formulation:
D: given depth map. Q: credibility map. Guiding intensity image.
Garcia et al., “Pixel Weighted Average Strategy for Depth Sensor Data Fusion”, ICIP 2010
*credibility: 믿을 수 있음 , 진실성 ; 신용 , 신뢰성 , 신빙성 , 위신
192012-11-14 / Computer Vision Laboratory Seminar
Depth enhancement technique by sensor fusion
High resolutionimage
Low resolutiondepth
JBU result PWAS result
202012-11-14 / Computer Vision Laboratory Seminar
Depth enhancement technique by sensor fusion
Again, is PWAS ideal enough? Limitations of PWAS:
Degree of smoothing depends on gradient of low resolution depth map
Remaining problems: Degree of smoothing depends on gradients of pixels in depth map
Erroneous depths around depth edge are not compensated well
Contradictive with spatial weight term (fS(.))
Texture copy issue still remains in homogeneous regions of depth map
High-resolution guidance image(red=non-visible depth discontinu-
ities)
JBU enhanced depth map(zoomed)
PWAS enhanced depthmap (zoomed)
212012-11-14 / Computer Vision Laboratory Seminar
Depth enhancement technique by sensor fusion
Unified Multi-Lateral (UML) filter In order to reduce texture copy issue, the same author
proposed combined version of two PWAS F. Garcia, proposed in AVSS 2011
[Garcia2011] solution: Use of combined PWAS filters
The second filter has both spatial and range kernels acting onto D
Use of the credibility map Q as a blending function, i.e., β = Q
Representative formulation:
Depth pixels with high reliability are not influenced by the 2-D data avoiding texture copying
Garcia et al., “A New Multilateral Filter for Real-Time Depth Enhancement”, AVSS 2011
222012-11-14 / Computer Vision Laboratory Seminar
Depth enhancement technique by sensor fusion
Depth map enhancement examples
2D guidance image JBU PWAS UML
2D guidance image JBU PWAS UML
232012-11-14 / Computer Vision Laboratory Seminar
Depth enhancement technique by sensor fusion
Again, is UML ideal enough? Limitations of UML:
Features of the proposed filter strongly depends on the credibility map If reference pixel value in credibility map is low,
The filter works as the normal PWAS filter by in order to reduce edge blurring artifact by weakening smoothing effect around depth edge.
If reference pixel value in credibility map is high,
Relatively high weigh is allocated to J3, and the proposed filter works in direction of reducing texture copy artifact.
Remaining problems: Is credibility map really credible?
It only considers depth gradient, but occlusion, shadowing, and homogeneous regions are really incredible in general depth data.
Edge blurring artifact still exists when there’s no corresponding depth edge in the image due to similar object
colors.
242012-11-14 / Computer Vision Laboratory Seminar
Depth enhancement technique by sensor fusion
Depth map enhancement examples
Ground truth Downsampled (9x) Intensity image
JBU PWAS UML
252012-11-14 / Computer Vision Laboratory Seminar
Depth enhancement technique by sensor fusion
Generalized depth enhancement filter by sensor fusion
Generalize the previous UML filter not only for active sensors (RGB-D) but also more traditional stereo camera. F. Garcia, proposed in ECCV 2012
[Garcia2012] solution: Passive sensor: extension of credibility map for general depth data
Object boundary, occlusion, homogeneous regions are considered
Active sensor: adaptive blending function β(p) change to cope with edge blurring issue, and the second term (J3(p)) in UML is substituted by D(p)
Representative formulation:
Smoothing effect is reduced in credible depth regions The same computational with PWAS complexity New β(p) prevents edge blurring when image edges have similar color
Garcia et al., “Generalized depth enhancement filter by sensor fusion”, ECCV 2012
262012-11-14 / Computer Vision Laboratory Seminar
Depth enhancement technique by sensor fusion
Generalized depth enhancement filter by sensor fusion
Formulation of a new credibility map (Q(p):
Boundary map Qb(p) Qb(p) = Q(p) in J2
Occlusion map Qo(p):
Homogeneous map Qh(p): the characteristics of correlation cost at each pixel is analyzed
Homogeneous region flat correlation cost / repetitive pattern multiple minima. cost
First minimum value at depth d1 C(p, d1) / second minimum at d2 C(p, d2)
left/right consistency check
Garcia et al., “Generalized depth enhancement filter by sensor fusion”, ECCV 2012
272012-11-14 / Computer Vision Laboratory Seminar
Depth enhancement technique by sensor fusion
Generalized depth enhancement filter by sensor fusion
Formulation of blending function β(p):
QI is defined analogously to QD but considering I∇
The function u(·) is a step function If edge blurring condition is satisfied, β(p) = 1
i.e., QD < τD (QD = Q, defined in (PWAS)), and QI > τI
The constants τI and τD are empirically chosen thresholds
If not, β(p) = QD(p), and J5(p) works similarly to the conventional UML filter
Garcia et al., “Generalized depth enhancement filter by sensor fusion”, ECCV 2012
282012-11-14 / Computer Vision Laboratory Seminar
Depth enhancement technique by sensor fusion
Experimental results – passive sensing
Garcia et al., “Generalized depth enhancement filter by sensor fusion”, ECCV 2012
292012-11-14 / Computer Vision Laboratory Seminar
Depth enhancement technique by sensor fusion
RMS: Root Mean Square
PBMP : Percentage of Bad Matching Pixels
SSIM : Structural SIMilarity
Experimental results – passive sensing
Garcia et al., “Generalized depth enhancement filter by sensor fusion”, ECCV 2012
302012-11-14 / Computer Vision Laboratory Seminar
Depth enhancement technique by sensor fusion
Experimental results – active sensing
Image UI
QD β
Garcia et al., “Generalized depth enhancement filter by sensor fusion”, ECCV 2012
312012-11-14 / Computer Vision Laboratory Seminar
Depth enhancement technique by sensor fusion
Experimental results – active sensing
Garcia et al., “Generalized depth enhancement filter by sensor fusion”, ECCV 2012
322012-11-14 / Computer Vision Laboratory Seminar
Depth enhancement technique by sensor fusion
Now then, do we have an optimal solution? Limitations:
Initial depth 의 신뢰도가 낮을 경우에 image value 의 문제가 있으면 해당 position 의 depth 를 개선할 방법이 없다 . 예를 들어 , occlusion, homogeneous 영역에서 texture copying 문제가 여전히
발생 가능함 . UML filter 컨셉과 충동 ! Edge blurring 조건에서 depth edge 근처의 distortion 확산의 문제
Remaining problems: Qb, Qo, Qh 의 역할이 완전히 독립적이지 못하기 때문에 over weighting 의
위험이 우려됨 . 예를 들어 , boundary 와 occlusion 영역이 겹치게 된다 . 혹은 homogeneous 영역에서 잘못 추정된 depth 는 left/right consistency 결과
occlusion 영역으로 판단될 수도 있다 .
332012-11-14 / Computer Vision Laboratory Seminar
Depth enhancement technique by sensor fusion
Conclusion Joint bilateral upsampling approach
Propagates properties from one to an other modality Credibility map decides system performance Defining blending function can be another critical factor Many empirical parameters make the practical automated usage
of such fusion filter challenging Another question is a clear rule on when a smoothing by filtering
is to be avoided and when a simple binary decision is to be undertaken