When Does Computational
Imaging Improve Performance?
Oliver Cossairt
Assistant Professor
Northwestern University
Collaborators: Mohit Gupta, Changyin Zhou,
Daniel Miau, Shree Nayar (Columbia University)
How to capture the best quality photograph?
Iphone 4GPhotograph
Canon DLSRPhotograph
http://campl.us/posts/iPhone-Camera-Comparison
http://thehdrimage.com/tag/metering/
Digital vs. Film Photography
• Dynamic range fixed at time of exposure
1ms Exposure Time 4ms Exposure Time
8ms Exposure Time 16ms Exposure Time
Digital Camera
• Dynamic range can be extended computationally
High Dynamic Range Image
Film Camera
http://graphics.stanford.edu/courses/cs178-13/lectures/astrophotography-02may13-150dpi.pdf
Digital Photography: Noise
• Noise level fixed at capture time
• Limited by film grain size
Milky Way, 6 min exposure (Jesse Levinson)
Digital Camera• Noise can be
averaged away
• SNR unlimited in principle
Milky Way, 10x 6 min exposures averaged
Film Camera
• Take multiple pictures and computationally combine
Computational Imaging: Increased Functionality
Others• Multiview Stereo• Depth from Focus/Defocus• Tomography• Structured Light• Deconvolution microscopy• etc.
HDR Imaging Panoramic Stitching
Digital Holography
[Greenbum et al. ’12]
Image-Based Lighting
[Debevec et al. ’00]
Light Field Capture
[Wilburn et al. ’04]
[Caraso, Opt. Eng ‘06]
Focus Blur
• Blur is fixed at time of capture
M100 Galaxy captured by Hubble Telescope
Digital Camera
• Images can be deblurredvia deconvolution
M100 Galaxy after blind deconvolution
Film Camera
Digital Camera
• Long exposure produces blurry image
50 millisecond exposure
Motion Blur
Film Camera
• Short exposure avoids motion blur
• Image is noisy
1 millisecond exposure (noisy)
• Blur can be removed via deconvolution
(blurry)(deblurred)
Coded Blur and Multiplexing
CameraExposure
Time50 millisec
[Raskar et al. ’06]
Coded Blur and Multiplexing
CameraExposure
Time50 millisec
[Raskar et al. ’06]
Blur is shifted and summed copies
Coded Blur and Multiplexing
CameraExposure
Time50 millisec
[Raskar et al. ’06]
Which copies to keep?
Coded image capture for increased performance
Computational Imaging: Increased Performance
Reflectance
[Schechner ‘03][Ratner ‘07][Ratner ‘08]
Defocus Blur
[Hausler ’72][Nagahara ’08]
[Levin et al. ’07][Zhou, Nayar ’08]
[Dowski, Cathey ‘96]
Motion Blur
[Raskar ’06] [Levin ‘08][Cho ’10]
Multi/Hyper-Spectral
[Sloane ’79] [Hanley ’99]
[Baer ‘99][Wetzstein et al., ’12]
Light Field Capture
[Lanman ’08] [Veeraraghavan ’07]
[Liang ‘08]
Coded Aperture
[Mertz ’65] [Gottesman ’89]
Coded Imaging Performance
CameraExposure
Time50 millisec
Coded Exposure
Time50 millisec
Short Exposure
Vs.
Deblurred Image
When does computational imaging improve performance?
CameraExposure
Measuring Computational Imaging
Performance
• No diffraction
• Fully determined
Image Formation Model
Assumption:
A) Linear model of incoherent image formation
Scene CodedImage
ComputationalCamera
Image
Optical Coding Equation
CodedImage
CodingMatrix Noise
• Signal dependent / independent noise
• Ignore Dark current, fixed pattern
Affine Noise Model
Assumption:
B) Affine noise model (photon noise is Gaussian)
Noise Variance at kth Pixel:
photon noise
aperture, lighting, pixel size
read noise
electronics,ADC’s, quantization
• Photon noise modeled as Gaussian
(ok for more than 10 photons)
• Photon noise spatially averaged
Noise PDF:
Signal-level and photon noise depend on illumination
Lighting Conditions
Assumption:
C) Naturally occurring light conditions for photography
Ex) q=.5, R = .5, F/8, t = 6ms, p=6um
Isrc (lux)
Quarter moon Full moon Twilight Indoor lighting Cloudy day Sunny day
10-2 1 10 102 103 104
8×10-3 0.8 8.1 81.4 814.3 8143J (e-)
Signal level
Illumination
Illumination(lux)
Aperture ExposureTime (s)
PixelSize (m)
QuantumEfficiency
ReflectivityAverageSignal (e-)
[Cossairt et al. TIP ‘12]
For Gaussian noise, Mean-Squared-Error (MSE) can be computed
analytically
Measuring Performance
Observation:
1) Multiplexing performance depends on coding matrix
Ex) Coded Motion Deblurring
Long Exposure Coded Exposure
Impulse imaging (identity sampling)
Multiplexing vs. Impulse Imaging
Observation:
2) Multiplexing increases signal-dependent noise
Noise variance
Coded imaging (multiplexed sampling)
Noise variance
Increasedthroughput
Observation:
3) Performance depends on multiplexing and signal prior
Multiplexing vs. Impulse Imaging
SNR Gain over impulse imaging:
Decreases with C
Noise Dependent
Increases with C
Coding Dependent
Coded Aperture Astronomy
Decreasing contrast
Increasing scene points
[Mertz ’65]
Fresnel zone plate
[Sloane ’79]
Hadamard Multiplexing:
No SNR gain for large signal
Assume we have a PDF for images, e.g.
Image Prior Models
Assumption:
D) Signal prior models naturally occurring images
MSE difficult to express analytically when
Compute the Maximum A Posteriori (MAP) estimate
Data term Prior term
Other priors
•Total Variation (TV)
•Wavelet/sparsity prior
•Learned priors (K-SVD)
Power Spectra Prior
Image Priors and Noise
Observation:
4) Signal priors help more at low light levels
PSNR = 5.5 dB
Twilight
(10 lux)
PSNR = 16.4 dB
Denoise
Daylight
(10 lux)5
PSNR = 35 dB PSNR = 35.9 dB
Denoise
How to capture the best quality photograph?
Observations:
1) Multiplexing performance depends on coding matrix
2) Multiplexing helps most in low light
3) Performance depends on both multiplexing and signal prior
4) Signal priors help most in low light
Assumptions
A) Incoherent imaging
B) Affine noise model
C) Natural lighting conditions
D) Natural image prior
Example:
Motion Deblurring
Motion Deblurring vs. Impulse Imaging
CameraExposure
Time50 millisec
Computational Imaging (Coded Exposure)
Time50 millisec
Impulse Imaging (Short Exposure)
Vs.
What is the best possible coding performance we can get?
Optical efficiency (C)= total ‘on’ time
[Ratner ‘07]
[Ratner and Schechner ‘07]
Optical Efficiency (C)
SN
R G
ain
(G
)
Average signal level
Read noise
5 10 15 20 25 30
1
2
3
4
S-Matrix
Multiplexing Performance Bound
When Does Motion Deblurring Improve Performance?
Motion InvariantLevin et al.
Flutter ShutterRaskar et al.
Upper Bound on SNR Gain:
Performance depends only on lighting conditions!
[Cossairt et al. TIP ‘12]
Read
Noise
Average
signal level
q=.5, R = .5, F/2.1, p = 1um,
Maximum object speed (pixels/sec)
Twilight
(10 lux)
Flutter Shutter Simulation
Impulse (4ms) Flutter Shutter (180ms) Deblurred
Cloudy Day
(10 lux)3
PSNR = 12.4 dB PSNR = 10.1 dB
q=.5, R = .5, F/2.1, pixel size = 1um, read noise
PSNR = -7.2 dB PSNR = -3.0 dB
Example:
Extended DOF Imaging
http://en.wikipedia.org/wiki/Focus_stacking
Depth of Field
Microscope Tachinid Fly
Small DOF
http://en.wikipedia.org/wiki/Focus_stacking
Depth of Field
Microscope Tachinid Fly
Large DOF
F 1.4F 2.8F 5.6F 8.0
LensImage
Depth of Field and Noise
Small apertures have large depth of field and low SNR
LensSensor
Focal Sweep
400 600 900 1500 20001200 1700 (depth)
Point Spread Function (PSF)
[Hausler ‘72, Nagahara et al. ‘08]
Focal Sweep
400 600 900 1500 2000 (depth)1200 1700
+ + + + + + =
t = 1 t = 2 t = 3 t = 4 t = 5 t = 6 t = 7(400) (600) (900) (1500) (2000)(1200) (1700)
Integrated PSF
[Hausler ‘72, Nagahara et al. ‘08]
LensSensor
Quasi Depth Invariant PSF
Extended depth of field with a single deconvolution
1200mm
750mm
400mm2000mm
-25 0 25
0.016
0.012
0.008
0.004
0
1
60
1200mm
-25 0 25
0.016
0.012
0.008
0.004
0
750mm
400mm 2000mm
×
Traditional Camera PSFFocal Sweep PSF
Focal Sweep: Captured
75 m
50 m
Extended Depth of Field Telescope
Meade LX200 8’’ Telescope
Traditional Image
75 m
50 m
75 m
50 m
Focal Sweep: Processed2000mm FL
LensImage
Focal Sweep Without Moving Parts
Focal Sweep
LensImage
Diffusion Coding
Radial Diffuser
(No Moving Parts)
[Cossairt et al. Siggraph ‘10]
500 x 3 micron
Diffusion Coding: Evaluation
RM
S D
eblu
rrin
gErr
or
Depth (mm)
Cubic Phase PlateFocal Sweep
noise
Diffusion Coding
400 600 900 1200 1500 1700 2000
.07
.06
.05
.04
.03
.02
.01
[Dowski and Cathey ‘95]
[Hausler ‘72, Nagahara et al. ‘08]
[Cossairt et al. Siggraph ‘10]
Diffusion coding gives best performance without moving parts
Diffusion Coding F/1.8 (Deblurred)
Traditional F/1.8 Diffusion Coding F/1.8 (Captured)
Diffusion Coding vs. Traditional Camera
Traditional F/18 (Normalized)
Face Detection
Diffusion Coding Camera (F/2.0)Traditional Camera (F/2.0)
AnnularAperture
Mirror 2 Mirror 1 SensorDiffuser
80” Focal Length
8” dia
Diffusion Coded Telescope: Optical Design
Telephoto Focal Sweep with Deformable Optics
Canon 800mm EFL Lens
Deformable Lens
Sensor
[Miau et al. ICCP ‘13]
Telephoto Video Quality Comparison
Conventional EDOF (Deformable Lens)
Noise Variance:Impulse Camera
lenssensor
A
Focal Sweep Performance
Mean-Squared Error:
Noise Variance:
light increase
Focal Sweep
lens
diffuser
sensor
C*A
Mean-Squared Error:
[Cossairt et al. TIP ‘12]
When Does Defocus Deblurring Improve Performance?
Performance depends only on lighting conditions!
[Cossairt et al. TIP ‘12]
Read
Noise
Average
signal level
Focal sweep multiplexing gain can be expressed analytically
q=.5, R = .5, t = 20ms, p = 5um,
Maximum defocus at F/1 (pixels)
Focal Sweep
(F/2.0)
Traditional
(F/2.0)
Twilight
(10 lux)
Pixel size = 5um
Read noise
Focal Sweep Simulation
Daylight
(10 lux)5
PSNR = 35 db PSNR = 38.5 db
Traditional
(F/20.0)
PSNR = 5.5 db PSNR = 18.5 db
Focal Sweep
(F/2.0)
Focal Sweep Simulation (with Prior)
Daylight
(10 lux)5
PSNR = 35.9 dB / 35 dB PSNR = 39.6 dB / 38.5 dB
BM3D Algorithm: [Dabov et al. ‘06]
Twilight
(10 lux)
Traditional
(F/20.0)
PSNR = 16.4 dB / 5.5 dB PSNR = 22.8 dB / 18.5 dB
Traditional
(F/2.0)
Pixel size = 5um
Read noise
Simulated Focal Sweep Performance
[Cossairt et al., TIP ‘12]
Focal Sweep performance bound is weak at low light levels
Focal Sweep Performance
ImpulseImaging
Conclusions
• Results for Motion Deblurring, EDOF also applicable to many
other computational cameras
• Computational imaging performance should always be
measured relative to impulse imaging
• Computational imaging performance depends jointly on
multiplexing, noise, and signal priors
• Important question: “How much performance improvement
from multiplexing above and beyond use of signal priors?”
Visual Quality Metrics
[Cossairt et al., TIP ‘12]
SSIM Metric UQI Metric SSIM MetricVIF Metric
Performance bound roughly holds for all metrics
Computational Gigapixel Camera
BallLens
LensArray
BallLens
SensorArray
Computations
GigapixelImage
Also See: MOSAIC Program, Duke, UCSD, Distant Focus
Ball LensSensor
Pan/Tilt Motor
Computational Camera Design Prototype Camera
Resolution vs. Lens Scale
Pixels
Poin
t Spre
ad F
unction
0-50 50
PSF size increases linearly
Resolution vs. Lens Scale
Pixels
Poin
t Spre
ad F
unction
0-50 50
PSF size increases linearly
Scale (M)
RM
S D
eblu
rrin
gErr
or
1 12 25 37 50 62 75 87 100
0.02
0.04
0.06
0.08
0.10
0.12
0.14
Deblurring Error is sub-linear
[Cossairt et al. JOSA ‘11]
