2015.07.10.
BME IIT CG
3D reconstruction
with depth image fusion
Szirmay-Kalos László
WAIT 2015 BME IIT CG
Depth (range) cameras
SSIP
WAIT 2015 BME IIT CG
Application: limitations of compositingChroma keying
Augmented reality
Compositing can be based on color:
• Fixed order
• No shadows
• No reflections, refractions, cross illumination
SSIP WAIT 2015 BME IIT CG
Depth compositing (Zinemath)
SSIP
2015.07.10.
WAIT 2015 BME IIT CG
3D reconstruction of a point
t
=***
***
***
zzz
yyy
xxx
kji
kji
kji
R
tRpp += cw
f
(xc,yc,zc)
xi = fxc/zc
yi = fyc/zc
(cx, cy)(cx,cy)
(X,Y)
xi =(X-cx)sx
yi =(Y-cy)sy
(xi,yi)
zc
tdtRp +=+
−
−
= ccyy
xx
w zYXzscYf
scXf
),(
1
)(
)(
SSIP
Back projection
WAIT 2015 BME IIT CG
Stereo vision and active light
111 tdp += cw z222 tdp += cw z
111 tdp += cw z
222111 tdtdp +=+= ccw zz
SSIP
WAIT 2015 BME IIT CG
Time of flight sensors
Pulsed modulation:
Accurate time measurement
expensive
Continuous modulation
Periodic distance
SSIP WAIT 2015 BME IIT CG
Temporal noise filtering and back
projection for static camera
R, t
SSIP
2015.07.10.
WAIT 2015 BME IIT CG
3D triangle mesh: height field
Topology is a
height field
R, t
SSIP WAIT 2015 BME IIT CG
Linear and Bilateral filtering
∑∑∑ +=
+= )(|)(|
|)(|
)(|)(|)(
~opo
o
opop Dw
w
DwD
∑∑
−+
+−+=
|))()((||)(|
)(|))()((||)(|)(
~
popo
oppopop
DDww
DDDwwD
is
is
∑∑
•+
+•+=
),,(
)(),,()(
~
opo
opopop
r
DrD
Linear filter:
Bilateral filter:
Generalization:
|))()((||))()((||)(|),,( pIopIpopoopo −+−+=•+ cds wDDwwr
SSIP
WAIT 2015 BME IIT CG
Moving objects: optical flow
p
p + v
t
t+1
Classical optical flow constraint:
Gradient constancy:
)1,(),(),( ++−= ttEc vpIpIvp
)1,(),(),( ++−= ttEg vpIp
pIp
vpd
d
d
d
Smoothness:22
,,,,),(tyxtyx
Eyyyxxx
s ∂
∂
∂
∂
∂
∂+
∂∂
∂∂
∂∂
=vvvvvv
vp
Energy functional:
pvpvpvpv d),(),(),()( sgc EEEE βα ++= ∫
SSIP WAIT 2015 BME IIT CG
Spatio-Temporal filtering
p*=p - v
p*
q=p+o
q*
*),(
)**,(
|)*)()((|
|)*)()((|
*)*,,,(
qp
qp
qIpI
qpqp
d
s
d
c
w
w
DDw
w
r
−
−
=
p
t
t+1
q*
p*
q*
q
p
SSIP
p
2015.07.10.
WAIT 2015 BME IIT CG
System
RGB
camera
depth
camera
Lens distortion
compensation
3D
mesh
Lens distortion
compensation
R, t
reprojection
+ upsampling
Optical
flow
Spatio-
temporal
filtering
Filtered
Point cloud
Rendering
with OpenGL
GLSL
shaders
WAIT 2015 BME IIT CG
Results
SSIP
WAIT 2015 BME IIT CG
Dynamic camera, static scene
FusionPoint cloud
Ob
jectProjection
Problems:
• in different images the camera changes
camera tracking based on static objects
• in different frames different points are visible
We need to maintain surface information
between points
SSIP
Solution (Curless/Levoy):
• Scene is represented by an emerging
distance field
WAIT 2015 BME IIT CG
Distance field (signed, truncated)
d
SSIP
d(x,y,z)
voxel
2015.07.10.
WAIT 2015 BME IIT CG
Rendering distance fields: ray marching
normal = grad d
d(x,y,z) < 0
d(x,y,z) > 0
SSIP
d(x,y,z) t
t
WAIT 2015 BME IIT CG
Ray marching on GPUfor(t = 0; t < exit; t += dt) {
float3 q = eye + raydir * t;
if (tex3D(volume, q) < 0) {
float3 normal = float3( tex3d(volume, q + float3(1/RES,0,0)) –
tex3d(volume, q - float3(1/RES,0,0)),
tex3d(volume, q + float3(0,1/RES,0)) –
tex3d(volume, q - float3(0,1/RES,0)),
tex3d(volume, q + float3(0,0,1/RES)) –
tex3d(volume, q - float3(0,0,1/RES)));
normal = normalize( normal );
color = lightint * kd * max(dot(lightdir, normal), 0);
distance = t;
return;
}
}
color = float3(0,0,0);
distance = -1;
return;
SSIP
WAIT 2015 BME IIT CG
Surface generation: marching cubes
d(x,y,z) > 0
d(x,y,z) < 0
0
1
4
2
3
5
6
7
Case: 000011102 =14
2; 0-1; 0-3; 2-6; 0-3; 3-7; 2-6 14
0;
14
0
255 0;
SSIP WAIT 2015 BME IIT CG
Volumetric fusion
1 0.9 0.5 -0.1 -0.8 -1
0.5
SSIP
2015.07.10.
WAIT 2015 BME IIT CG
Camera tracking: Iterative Closest Point
Backprojection
of current
image
Zero crossings of
the 3D distance
field
q1
q2
qnR, t
p1p2
pn
min1
)( →−⋅−= ∑i
iin
E tqRptR,
SSIP
Projective data association
WAIT 2015 BME IIT CG
System
3D distance field3D
mesh
Depth
To Distance
Volumetric
FusionRay
marching
ICP
Camera track
(PCL library)
Ne
w d
ep
th i
ma
ge
R, t
Depth + normal image
Point cloud
and mesh
generation
• Volumetric model: 128 x 128 x 128 +
• Image resolution: 640 x 480
• Speed: 30 FPS
• Hardware: NVIDIA 690 GT, CUDA
SSIP
WAIT 2015 BME IIT CG
Result on synthetic data
SSIP WAIT 2015 BME IIT CG
Additional filtering of distance fields
2015.07.10.
WAIT 2015 BME IIT CG
Additional filtering of distance field
WAIT 2015 BME IIT CG
Implementation and
Conclusions
• With GPU real-time 3D reconstruction is possible
• Height field representation:
–Simplified implementation: http://3d-scene.org/
• Volumetric reconstruction:
–Public domain simplified implementation: PCL library
–Commercial: KinectFusion
SSIP