Marc Levoy
Using Plane + Parallax to Calibrate Dense Camera Arrays
Vaibhav Vaish, Bennett Wilburn, Neel Joshi, Marc Levoy
Computer Science DepartmentStanford University
Marc Levoy
The Stanford Multi-Camera Array
Marc Levoy
Synthetic Aperture Photography: Seeing through Foliage
Marc Levoy
Synthetic Aperture Photography: Seeing through Foliage
Marc Levoy
Outline
• Problem Statement– Synthetic aperture photography using an array
of cameras
– Calibration required
• Calibration Pipeline
• Results
• Future Work
Marc Levoy
Synthetic aperture photography
Marc Levoy
SAP: Prior Work
• Synthetic Aperture Radar
• Light Field Rendering [Levoy 96]
• Dynamically Reparametrized Light Fields [Isaksen 00]
• Single lens SAP [Favaro 03]
Marc Levoy
Outdoor SAP: Array layout
• width of aperture 2m• number of cameras 45• spacing between cameras 13cm• camera’s field of view4.5°
Marc Levoy
Outdoor SAP: The scene
• distance to occluder 33m
• distance to targets 45m
• field of view at target 3m
Marc Levoy
Outdoor SAP: Calibration
• Narrow field of view and long-range imaging makes accurate pose estimation difficult• Cannot take calibration measurements at the desired focal depth (behind occluding
bushes)
Calibration Volume
Focal Depth
28 m 5m
a = 2m
12m
Marc Levoy
Calibration Goal
Given a focal plane, compute the projective transform (homography) to project each camera image onto the plane.
Focal Plane
Marc Levoy
Approaches to Calibration
• Metric Calibration– Computes camera intrinsics, position,
orientation(10 parameters/camera)
– Nonlinear optimization, requires initial guess– Not stable for narrow angle lenses and long
range imaging
• Non-metric Calibration– Plane + Parallax methods [Irani 96, Triggs 00]– Homography Spaces [Zelnik-Manor 99]
Marc Levoy
Calibration Pipeline
• Problem Statement
• Calibration Pipeline– Focus cameras on one plane (using homographies)
– Compute relative camera positions from parallax measurements
– Use camera positions to vary focal plane over a range of depths
• Results
• Future Work
Marc Levoy
Focusing on one plane
+
Add camera images so that points on one plane are in good focus
Marc Levoy
Focusing at different depths
+
Marc Levoy
Focusing at different depths
+
To focus at a different depth, we have to shift the images by an amount equal to the parallax
Marc Levoy
Parallax and Camera Geometry
p1= X1 . z/(Z0 + z) = X1 . dP
Parallax = Camera shift * Relative Depth
Reference Plane
Camera Plane
P
P
p1
X1
z
Z0
Marc Levoy
Parallax and Camera Geometry
Measure parallax of P in all cameras (wrt reference camera)
[ p1 p2 . . . ]T
Reference Plane
Camera Plane
p1
X1
P
p2
X2
Marc Levoy
Recovering Camera Positions
Parallax of point P:
For multiple points P1, … Pn :
Relative camera positions Xi can be recovered robustly (up to scale) using SVD.
Marc Levoy
Computing SAP Images at different focal depths
• To change the focal depth, images have to be shifted by the amount of parallax.
• In camera C1 , the parallax for a parallel focal plane is f . X1 ,where f is a constant that depends only on the depth of the plane.
• f is analogous to the focus distance of the synthetic lens: varying f changes the depth of the focal plane.
p1
X1
Marc Levoy
Algorithm for SAP
1. Focus cameras onto a frontoparallel plane
2. Compute parallax for one (or more) scene points
3. Recover relative camera positions Xi (up to an unknown scale)
4. For a range of values of f :• Shift image from camera Ci by f . Xi and average
shifted images.• Varying f corresponds to changing the focus
Marc Levoy
Results
Synthetic Aperture Sequence
Marc Levoy
Parallax v/s Metric Calibration
Parallax-based calibration Metric calibration
Marc Levoy
Summary
• Calibration of camera arrays for synthetic aperture photography
– Decompose warps into reference homography and shifts
– Use parallax measurements to compute camera positions
– Avoids computing camera intrinsics and orientation explicitly
– Robust, linear solution
• Metric information not available
• Algorithm requires planar camera array and frontoparallel reference plane
Marc Levoy
Extension :Tilted Focal Planes
Reference PlaneFocal Plane
e (epipole)
L (line of intersection)
- Parallax is described by a projective warp (not a shift)
- Rank-1 factorization is still possible
Marc Levoy
Future Work
• Real-time applications– Warp images in hardware– Track moving objects by moving focal plane
• 3D Reconstruction from synthetic focus– Is this more robust to occlusions ?
• Quantitative analysis of synthetic aperture photography
– Effect of occluder density, number of cameras, aperture shape
Marc Levoy
Acknowledgements
• Sponsors– NSF IIS-0219856-001
– DARPA NBCH 1030009
• Assistance in acquisition– Gaurav Garg, Augusto Roman, Billy Chen, Pradeep Sen,
Doantam Phan, Guillaume Poncin, Jeff Klingner
High Speed Videography Using a Dense Camera ArrayB Wilburn, N Joshi, V Vaish, M Levoy, M Horowitz
Session 5A, 4:20pm