Introduction to the Light-Field
Camera
Speaker: Wei-Sheng Lai
Date: 2013/11/29
Outline
1. Preface
2. Image Rendering
3. Light-Field History
4. Plenoptic 1.0
5. Plenoptic 2.0
6. Different Light-Field Camera
7. Conclusion
2
Preface β What is Light-Field Camera?
1. Website: https://www.lytro.com/
2. Property: Refocusing, change orientation, display photo
in 3D
3
Image Rendering
β’ A tradition image is formed by integrating rays from
every direction at each pixel.
4
πΌ π = π
π π, π ππ
π: position (β π 2)π: direction (β π 2)π π, π : radiance
π°(π): pixel value
Image rendering
β’ A tradition image is formed by integrating rays from
every direction at each pixel.
5
πΌ π, π =1
π2
π
π
π[π, π,π, π]
π, π: position
π,π: direction
π π, π,π, π : radiance
π°[π, π]: pixel value
Light-Field History
β’ The first light-field camera β 1908 Lippmannβs Integral
Photographs
6
Lippmannβs Integral Photographs Lippmann
Light-Field History
β’ 1968 Chutjian (First Digital Light-Field)
7
Light-Field History
β’ 1992 Adelson - Plenoptic camera
8[1] Adelson, E. H., Wang, J. Y. βSingle lens stereo with a plenoptic cameraβ. IEEE TPAMI, 14(2), 99-106, 1992.
Micro-lens
Main lens
β’ Main lens focused on micro-lens, micro-lens focused on
infinity
Plenoptic 1.0
9[1] Adelson, E. H., Wang, J. Y. βSingle lens stereo with a plenoptic cameraβ. IEEE TPAMI, 14(2), 99-106, 1992.
Plenoptic 1.0
β’ Raw data
10
Plenoptic 1.0
β’ Plenoptic image is a flat 2D array of 2D arrays
11
direction
position
Plenoptic 1.0 - Rendering
β’ Light-field rendering
12
Plenoptic 1.0 - Rendering
β’ Render different viewpoint
13
Plenoptic 1.0 - Rendering
β’ Render different viewpoint
14
Plenoptic 1.0 - Rendering
β’ Render different viewpoint
15
Plenoptic 1.0 - Rendering
β’ Render different viewpoint
16
Plenoptic 1.0 - Rendering
β’ Render different viewpoint
17
Plenoptic 1.0 - Rendering
β’ Render different viewpoint
18
Plenoptic 1.0 - Refocusing
β’ Change the distance from the lens to the sensor
19
sensor Main lens
Plenoptic 1.0 - Refocusing
β’ Change the distance from the lens to the sensor
20
sensor Main lens
Plenoptic 1.0 - Refocusing
β’ We capture radiance ππ. How to compute ππ ?
21
sensor Main lens
Micro-lens
Plenoptic 1.0 - Refocusing
β’ Ray travels distance π‘ through space
22
ππ ππ ray
β’ Refocusing is shearing transform: πβ² πβ², πβ² = π(π + π‘π, π)
πβ²
πβ²=1 π‘0 1
ππ =
π + π‘ππ
π: position (β π 2)π: direction/slope (β π 2 )
Plenoptic 1.0 - Refocusing
β’ Separate raw data into different viewpoint images
23ππ
ππ
ππ
ππ
ππππ
ππ
ππ
ππ
ππ
ππ
Plenoptic 1.0 - Refocusing
β’ Shearing in ππ₯ β merge
24
ππ
ππ
ππ
ππ
ππ
ππ
Plenoptic 1.0 - Refocusing
25
Plenoptic 1.0 - Refocusing
26
Plenoptic 1.0 - Refocusing
27
Plenoptic 1.0 - Refocusing
β’ Direct refocusing in the spatial domain requires π(π4)operations for each refocused image
β’ Ren Ng proposed the Fourier slice refocusing method:
28
Direct method
Initial
Refocusing
[2] Ng, Ren. "Fourier slice photography." ACM Transactions on Graphics (TOG). Vol. 24. No. 3. ACM, 2005.
Is Plenoptic 1.0 good enough?
β’ Micro-lenses images are defocused relative to main lens
image
β’ Low resolution
29
Plenoptic 2.0
β’ Lunsdaine and Georgiev 2008: Micro-lenses satisfy the
lens equation, and exactly focused on the main lens image.
30[3] Lumsdaine, Andrew, and Todor Georgiev. "Full resolution lightfield rendering."Indiana University and Adobe
Systems, Tech. Rep (2008).
Comparison
31
Comparison
32
Comparison
β’ Resolution analysis
33
pixel
Micro-lens
Plenoptic 2.0
β’ Rendering full aperture image
34
Plenoptic 2.0
β’ Rendering one view
35
Plenoptic 2.0
β’ Advantage of plenoptic 2.0:
1. Decouple resolution from the number of micro-lenses
2. Free to choose spatial-angular trade-off point
3. Each micro-lens is observing the scene as a slightly
shifted camera β good for super-resolution
36
37
38
39
40
Super-resolution
β’ Plenoptic 1.0 result
41
Other application
β’ We can put different filters on different micro-lenses
42
Different Light-Field Camera
β’ Lytro β the first hand-held light-field camera
β’ Plenoptic 1.0, resolution = 1080 x 1080 pixels (1.2M)
β’ Software can run both on MAC and Windows
43
Different Light-Field Camera
β’ Raytrix β target at industrial and scientific application
β’ Resolution from 1M to 3M
β’ Precise per-pixel depth information
44
Raytrix-R5 Raytrix-R11 Raytrix-R29 Raytrix-RX
Different Light-Field Camera
β’ Nokia Pelican Imaging β implement on smart phone
45[4] Venkataraman, Kartik, et al. "PiCam: an ultra-thin high performance monolithic camera array." ACM
Transactions on Graphics (TOG) 32.6 (2013): 166.
Conclusion
β’ What we have talked about:
1. The basic image rendering idea
2. The history of light-field camera
3. Property of Plenoptic camera 1.0
β’ Micro-lens focused on infinity
β’ Rendering and refocusing
4. Property of Plenoptic camera 2.0
β’ Micro-lens focused on main lens image
β’ Higher flexibility, suitable for super-resolution
5. Different kinds of light-field camera
46
47