IEEE 2017 Conference on
Computer Vision and Pattern
Recognition
Depth and Transient Imaging with Compressive SPAD Array CamerasQilin Sun Xiong Dun Yifan Peng Wolfgang Heidrich
KAUST KAUST KAUSTUBC&KAUST
Existing time-of-flight depth imaging and transient imaging systems are
limited either in terms of spatial&temporal resolution or are prohibitively
bulky&expensive.
Microlens Array
SPAD
Random Pattern
(a) Picosecond laser
(e) Reimaging
lens
(b) Imaging lens
(c) DMD
(d) TIR Prism
(f) SPAD array
𝑿 = arg min𝑿
1
2𝚿 𝑿 − 𝒀 𝟐
𝟐 +
𝑖
𝝀𝑖 𝑫𝑖(𝑿)
(a) Recovered data
(b) Raw data
SPADReimaging Lens
DLP4500
Imaging lens
HolderDMLA
Die SPAD
Sensor
START
END
DMD
Pattern
Ready?
Capturing
Done?
Control Signal
Patten
Data
Yes
Yes
Data
Stream
Control&Trigger
SynchronizingSPAD
Picosecond Laser
min𝐴,𝜇
𝑮 𝑡; 𝐴, 𝜇 𝚷(𝑡) − 𝒀𝟐
𝟐
Where ∈ 𝑅𝐾×𝑇×𝑛×𝑚 is the 4D data sharpened in the temporal domain
after modulation, 𝑿 ∈ 𝑅𝑇×𝑁×𝑀 is the 3D signal under evaluation, 𝚿 is an
operator that maps the random patterns to individual pixels at each layer,
and 𝑫 is 3D TV regularizer.
By jointly designing optics, mechanics, electronics, and computation, we
overcome the spatial resolution(64×32) limit of Single Photon Avalanche
Diode(SPAD) arrays by compressive sensing(image resolution up to
800×400) and realize a temporal resolution of ~20 picoseconds via a
physical temporal PSF model.
Where 𝒀 is the raw sensor data We present the sharpened sensor data 𝒀𝑖for each pixel 𝑖 as a sequence of Gaussians 𝑮.
As the picosecond laser pulse is approximately Gaussian and has a
FWHM ~80ps, the target Gaussian pulse is shown and denoted as 𝐆 who
has fixed 𝜎. Therefore, we can estimate the depth 𝜇 through solving a
least square problem.
Our SPAD array is working in TCSPC mode and the measurements from
each SPAD pixel could be reconstructed independently with tiling artifacts
addressed.
Light Concentrated by a lens
0ps
40ps
80ps
120ps
160ps
200ps
240ps
280ps
320ps
Scene
0ps
80ps
400ps
480ps
560ps160ps
240ps 640ps
320ps Scene
Transient Scene
Background and Our Solution
Models and Optimization
Prototype Working Flow
Resolution Analysis
Results
~8×+
50 40 30 20 10 0
Depth/mmM
ark
ed
Pix
els
profile
50 40 30 20 10 0
Depth/mm
depth intensity raw
0
20
40
60
Depth/mm
depth
intensity
raw
0
100
200
300
400
500
600
700
800
Compressive Depth Imaging
10mm
10mm
KAUST
*
0 1 2 3 Time/ns
200
400
0
Photo
n C
ounts
0 1 2 3 Time/ns
50
100
0
Photo
n C
ounts
0 1 2 3
0.5
1
0Time/ns Time/ns0 1 2 3
0.5
1
0
Low Pass
Filter
-1 =Target Gaussian Pulse
RC Response Model
at High Frequency
FWHM≈80ps
*~1ns
𝚷(𝑡)
Background noise
&Dark counts
200
400
0
Photo
n C
ounts
200
400
0
Photo
n C
ounts
Fitting Result
𝐆(t)
0 1 2 Time/ns 1 2 Time/ns03
Background noise
&Dark counts Removed
Optical Parameters in Experiment
E/F Mounts
TIR Prism
DMD
Bandpass Filter
Laser DMLA Auxiliary Optics
WaveLength 655nm Focal Length 1.035mm Imaging Canon 85mm Lens
Average Power ~1mW Structure2𝜋 period
24 Phase LevelReimaging
Inverted 0.9X EdmundDouble Side Telecentric
FWHM ~80ps Effiency(𝑓/20) 52.87% DMD TI DLP4500
Repetition Rate 50MHz FabricationLithography(0.7𝜇m)
+RIE etchingDMD optical
systemShown Below
𝛼1
𝛼2 𝜃𝑖1𝜃𝑜1
𝛼2 − 𝛼1
𝛾
𝜃𝑖2
𝜃𝑜2
𝛾 = 17.43°
SPAD Array Settings
Integration Time
HarwareBinning
Gate Width
Shiftper Cycle
PhotonsReceived
52𝜇s 1280 830ps 20ps ~40-60/pixel
Raw Reconstructed
