2012/11/11 1

High Quality Novel View Synthesis Based on Low Resolution Depth Image

and High Resolution Color Image

Jui-Chiu Chiang, Zheng-Feng Liu, and Wen-Nung Lie

National Chung Cheng University

Taiwan

2012/11/11 2

Outline Introduction Our hybrid camera system The proposed depth processing algorithm Experimental results Conclusion

2012/11/11 3

Color + Depth in 3DTV/FTV Depth image, in combination with DIBR (Depth

Image-Based Rendering) technique, is important in multi-view 3DTV/FTV applications.

DIBR

view8 view1view2view3view4view5view6view7view9

Color image Depth image

2012/11/11 4

Depth acquisition From stereo matching via disparity estimation

Unreliable for textureless regions Computationally intensive

From active photo-electrical sensing Lower resolution Higher acquisition speed and accuracy

Kinect SR4000

ToF

2012/11/11 5

Hybrid camera systems High-resolution color cameras + Low-resolution depth

camera (mis-alignment between their optical axis) [12]: Use warped data of depth camera to refine ROI

disparity of the dual-eye camera Proposed: one color image is used to refine the warped (to

both sides) and up-sampled depth images

SR4000

Camera L Camera R

2012/11/11 6

Processing diagram of our system

left color image

central depth image

right color image

Left preliminary

high-resolution depth

image processing

Right preliminary

high-resolution depth

image processing

Left high-quality

high-resolution depth

image processing

Right high-quality

high-resolution depth

image processing

Left depth image

Right depth image

2012/11/11 7

Generation of preliminary high-resolution depth image Purpose: to generate high-resolution depth image

at the left and right sides

Depth image from

depth camera

Post-processing warped

depths to remove incorrect

depth values

Preliminary high

resolution depth images

Pre-processing depths to

remove noise

Warping and up-sampling

depth image to the

reference views

Calibrated camera parameters

2012/11/11 8

Pre-processing of the depth map 33 median filter

Low resolution to high resolution depth warping Forward warping with a block size of 55 pixels

Post-processing by morphological operation Morphological opening

2012/11/11 9

Generation of high quality high-resolution depth image Color-image-guided depth map refinement

No

1

Segment n and Db(x,y)

depth value is correct?

depth value is correct?

0

depth value D(x,y) with Db(x,y)=1 inside the segment n is averaged and used as the updated depth value

depth value D(x,y) with Db(x,y)=0 inside the segment n is averaged and used as the updated depth value

No

Yes

Yes

depth value is unchanged

Sn depth value is unchanged

2012/11/11 10

Binarizing the warped depth image to have foreground/background classification

Color segmentation by mean-shift algorithm Label each segment as FG or BG

Refine the corresponding depth values according to the segment categorization

Bilateral filtering is performed for the edge parts of the modified depth image

2012/11/11 11

Experiments – system configurations Color image: 640x480 Depth image: 176x144 (SR4000) Intel Core 2 Duo Q6600 2.33 GHz, and DDR2

800 4GB

2012/11/11 12

Preliminary high-resolution depth image

(a) (c) (e)

(b) (d) (f)

After depth warping

After morphological operation

Traditional depth warping

2012/11/11 13

High-quality high-resolution depth image

(a) (c) (e)

(b) (d) (f)

Mean-shift

Incorrect pixels

After correction

Bilateral filtering

2012/11/11 14

A series of generated depth maps

2012/11/11 15

Virtual view synthesis based on various depth maps

(a) (c) (e)

(b) (d) (f)

Bilateral filtering

Bilateral filtering on edges

Bilateral filtering on edges + Gaussian

Hole filled

2012/11/11 16

Speed performance

ProcedureTime (sec)

Pre-processingDepth Warping 0.117

Color segmentation 0.742

Post-processingBilateral filtering on edges 0.352

Gaussian filtering 0.104

Novel view synthesis by DIBR (including hole filling) 0.144

2012/11/11 17

Conclusion A hybrid camera system consisting of high-

resolution color camera and low-resolution depth camera is proposed

A high-quality high-resolution depth images can be obtained for the left and right eyes – multi-view video plus depth (N=2 channels).

Future work Combine Kinect device and stereo camera to get better

depth image and cheaper cost