60
Achuta Kadambi MIT Media Lab 2 Macroscopic Interferometry Achuta Kadambi, MIT Media Lab Joint work with Jamie Schiel and Ramesh Raskar
Achuta Kadambi
MIT Media Lab
2
Macroscopic InterferometryAchuta Kadambi, MIT Media Lab
Joint work with Jamie Schiel and Ramesh Raskar
Achuta Kadambi
MIT Media Lab
3
Computer Vision/Graphics
Achuta Kadambi
MIT Media Lab
4
Computer Vision/Graphics Real-Life
Electric field only shown
700 nanometer wavelength
Achuta Kadambi
MIT Media Lab
5
Computer Vision/Graphics Real-Life
Electric field only shown
700 nanometer wavelength
Microscopic wave phenomena Macro scenes
Achuta Kadambi
MIT Media Lab
6
Multi-depth 3D Camera
Achuta Kadambi
MIT Media Lab
7
Multi-depth 3D Camera
10 meter scene
Achuta Kadambi
MIT Media Lab
8
Multi-depth 3D Camera
10 meter scene
Performance competitive low SNR
Achuta Kadambi
MIT Media Lab
9
Previous Work
ICCV15: Uses wave phenomena of polarization to enhance 3D shape
Kadambi et al. ICCV 2015
Achuta Kadambi
MIT Media Lab
10
Previous Work
ICCV15: Uses wave phenomena of polarization to enhance 3D shape
Today: Use wave phenomena to create multi-depth 3D
Kadambi et al. ICCV 2015
Achuta Kadambi
MIT Media Lab
11
Scene
A
BA
B
Achuta Kadambi
MIT Media Lab
12
Scene
A
BA
B
AI I
Achuta Kadambi
MIT Media Lab
13
Scene
A
BA
B
BI I
Achuta Kadambi
MIT Media Lab
14
Scene
A
BA
B
Achuta Kadambi
MIT Media Lab
15
Scene
A
BA
B
A BI I I
Achuta Kadambi
MIT Media Lab
16
Scene
A
BA
B
Achuta Kadambi
MIT Media Lab
17
Scene
A
BA
B
Interference Term
co2 sA B A BI II I I
Achuta Kadambi
MIT Media Lab
18
Scene
A
BA
B
Coherent Source
Interference graphic from sciencetalenter.dk
Interference Term
co2 sA B A BI II I I
Achuta Kadambi
MIT Media Lab
19
InterferometryIntentionally superimposing light to get scene clues
Figure from ligo.Caltech.edu
Achuta Kadambi
MIT Media Lab
20
Interferometry uses a reference to probe the scene
Reference Signal
Sample
Signal
Phase Delay
Achuta Kadambi
MIT Media Lab
21
Tiny Wavelength Good for Microscopic Scenes
Reference Signal
Sample
Signal
Phase Delay
700 nanometer wavelength
Achuta Kadambi
MIT Media Lab
22
Tiny Wavelength Good for Microscopic Scenes
Reference Signal
Sample
Signal
Phase Delay
700 nanometer wavelength
Corneal OCT
Corneal OCT from Carl Zeiss Visante ®
Achuta Kadambi
MIT Media Lab
23
Tiny Wavelength Good for Microscopic Scenes
Reference Signal
Sample
Signal
Phase Delay
700 nanometer wavelength
Corneal OCT
Corneal OCT from Carl Zeiss Visante ®
Achuta Kadambi
MIT Media Lab
24
Smaller the measurement ..
LIGO Project to Detect Grav Waves
Measure sample deviations less than the size of a proton
Achuta Kadambi
MIT Media Lab
25
Smaller the measurement .. Bigger the instrument
LIGO Project to Detect Grav Waves
Measure sample deviations less than the size of a proton
.. But kilometers in size
Achuta Kadambi
MIT Media Lab
26
Vision goal: small device, large scene
Achuta Kadambi
MIT Media Lab
27
Vision goal: small device, large scene
700 nanometer wavelength
Optical Phase
Achuta Kadambi
MIT Media Lab
28
Vision goal: small device, large scene
700 nanometer wavelength
Optical Phase Electronic Phase
Kinect uses electronic phase
Achuta Kadambi
MIT Media Lab
29
Terminology
Primal-Domain: Original Frame a Signal is Sampled In
Dual-Domain: Frequency transform with respect to Primal
Phase ToF: existing Kinect-style ToF cameras
Achuta Kadambi
MIT Media Lab
30
Phase ToF (Kinect)
Achuta Kadambi
MIT Media Lab
31
Phase ToF (Kinect)
2 Mz
c
f
Achuta Kadambi
MIT Media Lab
32
How the Phase ToF (Kinect) Works
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1Signal One and Signal Two
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 100000
1
2
3
4
5
6
7
8
9
10x 10
7 Fourier Transform of Cross Correlation Function dt = 0.0001;
N = 20000;
t = 0:dt:(N-1)*dt;
phi1 = pi/3; phi2 = 0;
s1 = sin(2*pi*1*t + phi1);
s2 = sin(2*pi*1*t+ phi2);
xcFFT = conj( fft(s1) ) .*
fft(s2);
[~, fundamental_idx] =
max(xcFFT);
phase_difference = angle(
xcFFT(fundamental_idx) );
Achuta Kadambi
MIT Media Lab
33
Block Diagram of Phase ToF
cos M ts t f
cos Mr t f t
Primal-domain is \tau
For clarity, all constants removed
Phase ToF Refer.
Phase ToF Sample
Achuta Kadambi
MIT Media Lab
34
Block Diagram of Phase ToF
Phase ToF Refer.
cos M ts t f
cos Mr t f t
For clarity, all constants removed
cos Mc f
Phase ToF Xcorr
Phase ToF Sample
Achuta Kadambi
MIT Media Lab
35
Compute FFT wrt.
& Take angle of fundam.
Block Diagram of Phase ToF
Phase ToF Refer.
Phase ToF Sample
cos M ts t f
cos Mr t f t
Primal-domain is \tau
For clarity, all constants removed
cos Mc f
Phase ToF Xcorr
Achuta Kadambi
MIT Media Lab
36
Problems with Phase ToF
Reflections of two or more sine waves come back and mix
Multipath Interference
Phase Wrapping
Phase is periodic – long-range depths will wrap
Low SNR
Calculating phase requires multiple steps (prop. of errors)
Achuta Kadambi
MIT Media Lab
37
Computational ToF Imaging
By count of recent papers, appears to be a hot area in comp. photo
[Heide and Hullin et al. 2013]
[Godbaz et al. 2008]
[Kadambi et al. 2013]
[Bhandari et al. 2014]
[Jayasuriya et al. 2015]
[Gkioulekas et al. 2015]
Find more details at ICCV 2015 course:
www.media.mit.edu/~achoo/iccvtoftutorial/
[Su et al. 2016] -- @ Wed AM posters
[Shreshtha et al. 2016] -- @ next month’s SIGGRAPH
[Tsai et al. 2016]
Achuta Kadambi
MIT Media Lab
38
Computational ToF Imaging
By count of recent papers, appears to be a hot area in comp. photo
[Heide and Hullin et al. 2013]
[Godbaz et al. 2008]
[Kadambi et al. 2013]
[Bhandari et al. 2014]
[Jayasuriya et al. 2015]
[Gkioulekas et al. 2015]
Find more details at ICCV 2015 course:
www.media.mit.edu/~achoo/iccvtoftutorial/
[Su et al. 2016] -- @ Wed AM posters
[Shreshtha et al. 2016] -- @ next month’s SIGGRAPH
[Tsai et al. 2016]
Our Contribution: change primal-domain
Achuta Kadambi
MIT Media Lab
39
Changing the Primal Domain
Compute FFT wrt.
Primal-domain is \tau
For clarity, all constants removed
cos Mc f
Phase ToF Xcorr
Phase ToF Refer.
Phase ToF Sample
cos M ts t f
cos Mr t f t
Achuta Kadambi
MIT Media Lab
40
Changing the Primal Domain
cos Mc f
Phase ToF Xcorr
For clarity, all constants removed
Achuta Kadambi
MIT Media Lab
41
Changing the Primal Domain
cos Mc f
Phase ToF Xcorr
For clarity, all constants removed
Achuta Kadambi
MIT Media Lab
42
Changing the Primal Domain
cos Mc f
Phase ToF Xcorr
For clarity, all constants removed
2 Mz
c
f
Achuta Kadambi
MIT Media Lab
43
Changing the Primal Domain
cos Mc f
Phase ToF Xcorr
For clarity, all constants removed
2 Mz
c
f
cos2
M Mc ffc
z
Achuta Kadambi
MIT Media Lab
44
Changing the Primal Domain
cos Mc f
Phase ToF Xcorr
For clarity, all constants removed
2 Mz
c
f
cos2
M Mc ffc
z
, cos2
M M M
zc f
cff
Achuta Kadambi
MIT Media Lab
45
Changing the Primal Domain
cos Mc f
Phase ToF Xcorr
For clarity, all constants removed
2 Mz
c
f
cos2
M Mc ffc
z
, cos2
M M M
zc f
cff
2
00, cosM M Mz
c ff fc
Achuta Kadambi
MIT Media Lab
46
Changing the Primal Domain
cos Mc f
Phase ToF Xcorr
For clarity, all constants removed
2 Mz
c
f
cos2
M Mc ffc
z
, cos2
M M M
zc f
cff
cos2
M M
zc f f
c
2
00, cosM M Mz
c ff fc
Achuta Kadambi
MIT Media Lab
47
Changing the Primal Domain
cos Mc f
Phase ToF Xcorr
For clarity, all constants removed
2 Mz
c
f
cos2
M Mc ffc
z
, cos2
M M M
zc f
cff
cos2
M M
zc f f
c
2
00, cosM M Mz
c ff fc
Frequency ToF Xcorr
Achuta Kadambi
MIT Media Lab
48
Changing the Primal Domain
cos Mc f
Phase ToF Xcorr
For clarity, all constants removed
2 Mz
c
f
cos2
M Mc ffc
z
, cos2
M M M
zc f
cff
cos2
M M
zc f f
c
2
00, cosM M Mz
c ff fc
Primal-domain is f_M
Dual-domain corresponds
to pathlength.
Frequency ToF Xcorr
Achuta Kadambi
MIT Media Lab
49
Achuta Kadambi
MIT Media Lab
50
Achuta Kadambi
MIT Media Lab
51
Macroscopic Scenes
Achuta Kadambi
MIT Media Lab
52
Macroscopic Scenes
Achuta Kadambi
MIT Media Lab
53
Macroscopic Scenes
Achuta Kadambi
MIT Media Lab
54
Multi-depth 3D Camera
Need only 4 frequencies to recover both reflections
Achuta Kadambi
MIT Media Lab
55
Multi-depth 3D Camera
Need only 4 frequencies to recover both reflections
Achuta Kadambi
MIT Media Lab
56
Multi-depth 3D Camera
Need only 4 frequencies to recover both reflections
Achuta Kadambi
MIT Media Lab
57
Resolving Multipath Interference
Scene
Achuta Kadambi
MIT Media Lab
58
How close can the reflections be?
Achuta Kadambi
MIT Media Lab
59
How close can the reflections be?
Achuta Kadambi
MIT Media Lab
60
Analyzing multipath estimation in low SNR
Achuta Kadambi
MIT Media Lab
61
Conclusion
Formalize link between ToF cameras and optical interferometry
Changed primal-domain for multi-depth, low SNR, long-range 3D
[email protected]/~achoo
Collaborators
Jamie SchielRamesh Raskar
Acknowledgements:
Ayush BhandariSuren JayasuriyaVage Taamazyan
mailto:[email protected]
