Date post: | 28-Dec-2015 |
Category: |
Documents |
Upload: | joan-hines |
View: | 215 times |
Download: | 0 times |
2012/11/11 1
High Quality Novel View Synthesis Based on Low Resolution Depth Image
and High Resolution Color Image
Jui-Chiu Chiang, Zheng-Feng Liu, and Wen-Nung Lie
National Chung Cheng University
Taiwan
2012/11/11 2
Outline Introduction Our hybrid camera system The proposed depth processing algorithm Experimental results Conclusion
2012/11/11 3
Color + Depth in 3DTV/FTV Depth image, in combination with DIBR (Depth
Image-Based Rendering) technique, is important in multi-view 3DTV/FTV applications.
DIBR
view8 view1view2view3view4view5view6view7view9
Color image Depth image
2012/11/11 4
Depth acquisition From stereo matching via disparity estimation
Unreliable for textureless regions Computationally intensive
From active photo-electrical sensing Lower resolution Higher acquisition speed and accuracy
Kinect SR4000
ToF
2012/11/11 5
Hybrid camera systems High-resolution color cameras + Low-resolution depth
camera (mis-alignment between their optical axis) [12]: Use warped data of depth camera to refine ROI
disparity of the dual-eye camera Proposed: one color image is used to refine the warped (to
both sides) and up-sampled depth images
SR4000
Camera L Camera R
2012/11/11 6
Processing diagram of our system
left color image
central depth image
right color image
Left preliminary
high-resolution depth
image processing
Right preliminary
high-resolution depth
image processing
Left high-quality
high-resolution depth
image processing
Right high-quality
high-resolution depth
image processing
Left depth image
Right depth image
2012/11/11 7
Generation of preliminary high-resolution depth image Purpose: to generate high-resolution depth image
at the left and right sides
Depth image from
depth camera
Post-processing warped
depths to remove incorrect
depth values
Preliminary high
resolution depth images
Pre-processing depths to
remove noise
Warping and up-sampling
depth image to the
reference views
Calibrated camera parameters
2012/11/11 8
Pre-processing of the depth map 33 median filter
Low resolution to high resolution depth warping Forward warping with a block size of 55 pixels
Post-processing by morphological operation Morphological opening
2012/11/11 9
Generation of high quality high-resolution depth image Color-image-guided depth map refinement
No
1
Segment n and Db(x,y)
depth value is correct?
depth value is correct?
0
depth value D(x,y) with Db(x,y)=1 inside the segment n is averaged and used as the updated depth value
depth value D(x,y) with Db(x,y)=0 inside the segment n is averaged and used as the updated depth value
No
Yes
Yes
depth value is unchanged
Sn depth value is unchanged
2012/11/11 10
Binarizing the warped depth image to have foreground/background classification
Color segmentation by mean-shift algorithm Label each segment as FG or BG
Refine the corresponding depth values according to the segment categorization
Bilateral filtering is performed for the edge parts of the modified depth image
2012/11/11 11
Experiments – system configurations Color image: 640x480 Depth image: 176x144 (SR4000) Intel Core 2 Duo Q6600 2.33 GHz, and DDR2
800 4GB
2012/11/11 12
Preliminary high-resolution depth image
(a) (c) (e)
(b) (d) (f)
After depth warping
After morphological operation
Traditional depth warping
2012/11/11 13
High-quality high-resolution depth image
(a) (c) (e)
(b) (d) (f)
Mean-shift
Incorrect pixels
After correction
Bilateral filtering
2012/11/11 14
A series of generated depth maps
2012/11/11 15
Virtual view synthesis based on various depth maps
(a) (c) (e)
(b) (d) (f)
Bilateral filtering
Bilateral filtering on edges
Bilateral filtering on edges + Gaussian
Hole filled
2012/11/11 16
Speed performance
ProcedureTime (sec)
Pre-processingDepth Warping 0.117
Color segmentation 0.742
Post-processingBilateral filtering on edges 0.352
Gaussian filtering 0.104
Novel view synthesis by DIBR (including hole filling) 0.144
2012/11/11 17
Conclusion A hybrid camera system consisting of high-
resolution color camera and low-resolution depth camera is proposed
A high-quality high-resolution depth images can be obtained for the left and right eyes – multi-view video plus depth (N=2 channels).
Future work Combine Kinect device and stereo camera to get better
depth image and cheaper cost