EE398A - Compression of Light Fields using 2-D Warping and Block Matching
1
Light Field Compression Using 2-D Warping and Block Matching
Shinjini KunduAnand Kamat Tarcar
EE398A Final Project
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
2
Outline
• Motivation and Goals• Overview of Our Method• Results and Analysis• Summary• Future Work• References
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
3
Motivation
• Light field images are used in computer graphics to compute new views of a scene without need for scene geometry model1.
• Need to compress large set of images• Exploit inter-view coherence to achieve
compression.
1. M. Levoy and P. Hanrahan, “Light field rendering,” in Computer Graphics (Proceedings SIGGRAPH 96), August 1996, pp. 31-42.
Light Fields• Represents a 3D scene or object from all viewing
positions and directions– 2D array of 2D images– Difficult to Acquire– Very Large
• Perfect representation requires images of the order of the resolution
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
5
Light Field Views
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
6
Light Field Data Set8.4 MB uncompressed data setshttp://lightfield.stanford.edu/aperture.swf?lightfield=data/lego_lf/preview.zip&zoom=1
Credit: Andrew Adams
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
7
Related Work• Intra-frame coding
– Vector quantization, DCT coding, transform coding yield compression ratios of less than 30:1
• Inter-frame coding (compression in the hundreds, thousands)– Disparity compensation– 3D geometry models– Blockwise
Compression ideal: maximally use coherence between two images
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
8
Our Method: 2-D Warping
• Each consecutive view is a projection of the previous view due to constant predictable movement of camera
• Find this relation between the views by obtaining projection matrix for each pair of views
• Predict the view and encode the residual
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
9
Our Encoding Scheme
Reconstructed Previous View
Previous Frame 2-D Warped
Lagrangian Cost
Function
Cost=R1+λD1
Cost=R2+λD2
2D Warping Algorithm
2-D DCT for the Residual
Residual and MV
?
Input View
--
Use for Reconstruction
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
10
Notes• DCT used on 8x8 blocks to encode residual• Laplacian distribution assumed for motion vectors• Projection matrix was encoded by normalizing values with
respect to 10, and assuming Laplacian distribution of bitrate. The min and max values are encoded separately using binary encoding.
• H =
-0.578 0.005 -0.720 -0.003 -0.572 0.007 0.000 0.000 -0.582
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
11
1. Feature match by correlation2. Projective matrix computedLagrangian Mode Decision using two references3. Clipped edges are interpolated using motion
compensation
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
12
Getting a predicted projection:Step 1: Feature matching by Correlation
corners detected corners detected
Features detected by Harris corner detection algorithm, and matching points identified by maximum correlation
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
13
Computing the Homography Matrix
• A homography is an invertible transformation from the real projective plane to the projective plane that maps straight lines to straight lines
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
14
Results for 2-D Projection Warping
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 131
32
33
34
35
36
37
38
Rate, in bits/pixel
PS
NR
(dB
)
Lego Men
motion compensation + DCT onlyprojection method
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
15
Results for 2-D Projective Warping
0 0.5 1 1.5 2 2.529.5
30
30.5
31
31.5
32Crystal Ball
Rate, in bits/pixel
PS
NR
(dB
)
data is for crystal ball light fieldmotion compensation + DCT onlyprojection method
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
16
Results for 2D Projective Warping
0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.529
30
31
32
33
34
35
36
37
38Lego Men vs. Crystal
PS
NR
(dB
)
Rate, in bits/pixel
Lego MenCrystal Ball
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
17
Compression Ratios
1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 60
50
100
150
200
250
300
350
400
450Compression Ratios
quantizer step size (log(Q))
com
pres
sion
ratio
projective methodmotion compensation only
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
18
Conclusion
• Advantages: decreased coding complexity, and increased rate/PSNR as well as compression
• Experimental results demonstrate improved coding efficiency with our 2D warp method when compared with MVC.
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
19
Future Work Possible
• Optimize the code to give better PSNR values and check performance by introducing extra modes like copy mode
• Explore other methods of using inter-view redundancy in detail like disparity compensation at sub-pel accuracy
• Run for larger data sets and optimize complexity of the algorithm
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
20
Summary
• Light fields represent a 3D scene using sequence of 2-D images
• Large amounts of data• Can use redundancy between images using 2-
D warping with motion compensated block matching
• Results in a sleek method for compression• Performance wise..
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
21
Acknowledgement
• Prof. Girod for pointing us in the right direction• Mina Makar for his help• Chuo-Ling Chang for DAPBT code• Huizhong Chen and Derek Pang for their help• Prof. Peter Kovesi for open source matlab
function library• Prof. Levoy’s group and Andrew Adams for
access to light field images
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
22
Questions?
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
23
Other Projects
• Use Motion Compensation with Directional Transforms
o Result: Gain in PSNR due to directionality is approximately 0.1dB at high Quantization; almost nil increase seen at low quantization
• So, We adapted the direction of out project to study a new approach of compression presented next.
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
24
Results with Motion Compensation and DAPBT for Crystal light field
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
25
Results with Motion Compensation and DAPBT for Lego light field
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
26
This is how blocking is done and direction selection happens!IAP(DAT)+IRP(DCT) for QP=44, Crystal Light Field
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
27
For Lego light field IAP(DAT)+IRP(DCT) for QP=44, Crystal Light Field