Sampling in image representation and compression
Alfredo Nava-Tudela Institute for Physical Science and Technology and Norbert
Wiener Center, University of Maryland, College Park
Joint work with John J. Benedetto Department of Mathematics and Norbert Wiener Center,
University of Maryland, College Park
Overview
• Problem statement • Image representation concepts • Image compression basics • Sparsity is the key, l0-minimization, OMP • Image compression revisited • Imagery metrics • Solving our problem: compressed sensing and
deterministic sampling masks • Solving our problem: results
9/24/13 Sampling in image representation and compression
1
9/24/13
Problem statement
2 Sampling in image representation and compression
9/24/13
Problem statement
3 Sampling in image representation and compression
9/24/13
Problem statement
JPEG, JPEG 2000
Sampling Compression
Representation
4 Sampling in image representation and compression
Image representation concepts
9/24/13 5 Sampling in image representation and compression
Image representation concepts
9/24/13
I[n1,n2]
0 ≤ n1 < N1, 0 ≤ n2 < N2
6 Sampling in image representation and compression
Image representation concepts
9/24/13
I[n1,n2] = pixel
n1
n2
7 Sampling in image representation and compression
Image representation concepts
9/24/13
I[n1,n2] ~ intensity, brightness at [n1,n2]
I[n1,n2] {0, … , 2B-1}, or I[n1,n2] {-2B-1, … , 2B-1-1}, where I[n1,n2] = round(2B I’[n1,n2]) and I’[n1,n2] [0,1) or [-½, ½)
€
∈
€
∈
€
∈
8 Sampling in image representation and compression
Image representation concepts
9/24/13 9 Sampling in image representation and compression
Image compression
9/24/13
512 x 512 x 8 x 3 = 6,291,456 bits
10 Sampling in image representation and compression
Image compression
9/24/13
JPEG, JPEG 2000
11 Sampling in image representation and compression
Image compression
9/24/13
1) Partitioning of the image I in sub-images
12 Sampling in image representation and compression
Image compression
9/24/13
1) Partitioning of the image I in sub-images 2) Transform sub-images to exploit
correlations within them
13 Sampling in image representation and compression
Image compression
9/24/13
1) Partitioning of the image I in sub-images 2) Transform sub-images to exploit
correlations within them 3) Quantize and encode
14 Sampling in image representation and compression
Image compression
9/24/13 15 Sampling in image representation and compression
Sparsity is the key
9/24/13
Cn u rd ths?
vs
Can you read this?
16 Sampling in image representation and compression
Sparsity is the key
9/24/13 17 Sampling in image representation and compression
9/24/13
Sparsity
The l0 “norm”:
||x||0 = # {k : xk ≠ 0}
18 Sampling in image representation and compression
9/24/13
l0-minimization ~ sparse solution
(P0): minx ||x||0 subject to ||Ax - b||2 = 0
19 Sampling in image representation and compression
9/24/13
l0-minimization ~ sparse solution
(P0ε): minx ||x||0 subject to ||Ax - b||2 < ε
20 Sampling in image representation and compression
9/24/13
l0-minimization ~ sparse solution
(P0ε): minx ||x||0 subject to ||Ax - b||2 < ε
Solving (P0 ε) is NP-hard!
Is there any hope?
21 Sampling in image representation and compression
Finding sparse solutions:OMP Orthogonal Matching Pursuit algorithm:
22
9/24/13
Finding sparse solutions:OMP Orthogonal Matching Pursuit algorithm:
23 Sampling in image representation and compression
9/24/13
Finding sparse solutions:OMP Orthogonal Matching Pursuit algorithm:
24 Sampling in image representation and compression
9/24/13
Finding sparse solutions:OMP Orthogonal Matching Pursuit algorithm:
25 Sampling in image representation and compression
Image compression
9/24/13 26 Sampling in image representation and compression
T = Tε = OMP(A, - ,ε), T’ = A
9/24/13
We need a matrix A
DCT Haar
27 Sampling in image representation and compression
9/24/13
We need a matrix A
2D - DCT 2D - Haar
28 Sampling in image representation and compression
9/24/13
We need a matrix A
29 Sampling in image representation and compression
9/24/13
Compressing a test image
30 Sampling in image representation and compression
c3() = b
= c3-1(b’)
x0 = Tε b = OMP(A, b ,ε)
b’ = T’ x0 = A x0
9/24/13
Compressing a test image
31 Sampling in image representation and compression
~ ? || b - b’ ||2 < ε
But what does that mean visually? How many bits were used?
9/24/13
Imagery metrics
Peak Signal-to-Noise Ratio (PSNR), measured in dB:
PSNR(X,Y) = 20 log10(MAXB / √MSE),
with MAXB = 2B-1, and MSE = ∑i,j [X(i,j) - Y(i,j)]2 /nm. In our case, n = m = 512, and B = 8, i.e. MAXB = 255.
32 Sampling in image representation and compression
9/24/13
Imagery metrics
Structural Similarity (SSIM), and Mean Structural Similarity(MSSIM) indices:
33 Sampling in image representation and compression
9/24/13
Imagery metrics
The normalized sparse bit-rate is
nsbr(I,A,ε) = ∑ ||xj||0/N1N2,
where image I is of size N1 by N2.
34 Sampling in image representation and compression
9/24/13
Imagery metrics: test images
Stream Boat
Elaine Barbara
35 Sampling in image representation and compression
9/24/13
Imagery metrics: bpp vs ε
36 Sampling in image representation and compression
9/24/13
Imagery metrics: bpp vs ε
37 Sampling in image representation and compression
9/24/13
Imagery metrics: bpp vs ε
38 Sampling in image representation and compression
9/24/13
Imagery metrics: bpp vs ε
39 Sampling in image representation and compression
9/24/13
Imagery metrics: bpp vs PSNR
40 Sampling in image representation and compression
9/24/13
Imagery metrics: bpp vs MSSIM
41 Sampling in image representation and compression
9/24/13
Imagery metrics: PSNR vs MSSIM
42 Sampling in image representation and compression
9/24/13
Compression results
ε = 32, c = 4 PSNR = 36.5220 dB, MSSIM = 0.9104, nsbr = 0.1609 bpp
43 Sampling in image representation and compression
Original Compressed SSIM
9/24/13
Back to our original problem
44 Sampling in image representation and compression
k = 40 (62.5%) k = 32 (50%)
9/24/13
Compressed sensing and sampling
45 Sampling in image representation and compression
minx ||x||0 subject to ||PA x – c ||2 < ε
P in Rk x n, A in Rn x m, and c in Rk
9/24/13
Deterministic sampling masks
46 Sampling in image representation and compression
ε = c , c = 4
€
k
9/24/13
Deterministic sampling masks
47 Sampling in image representation and compression
||A’ x’ – c ||2 < ε, with x’ = OMP(A’,c,ε), and x’ in Rm
9/24/13
Deterministic sampling masks
48 Sampling in image representation and compression
||A’ x’ – c ||2 < ε, with x’ = OMP(A’,c,ε), and x’ in Rm
= c3-1(A x’ )
9/24/13
Results
49 Sampling in image representation and compression
k = 40, c = 4 Luminance SSIM
9/24/13
Results
50
k = 40, c = 4
PSNR = 21.1575 PSNR = 39.7019 PSNR = 39.4193
9/24/13
Results
51 Sampling in image representation and compression
k = 40, c = 4 Deterministic sampling masks ~ Inpainting?
9/24/13
Results
52 Sampling in image representation and compression
k = 32, c = 4 PSNR = 29.8081 dB MSSIM = 0.7461
9/24/13
Results
53 Sampling in image representation and compression
k = 32, c = 4
9/24/13
Thank you!
54 Sampling in image representation and compression
9/24/13
References [1] A. M. Bruckstein, D. L. Donoho, and M. Elad, From sparse solutions of systems of equations to sparse modeling of signals and images, SIAM Review, 51 (2009), pp. 34–81."
[2] B. K. Natarajan, Sparse approximate solutions to linear systems, SIAM Journal on Computing, 24 (1995), pp. 227-234."
[3] G. W. Stewart, Introduction to Matrix Computations, Academic Press, 1973."
[4] T. Strohmer and R. W. Heath, Grassmanian frames with applications to coding and communication, Appl. Comput. Harmon. Anal., 14 (2004), pp. 257-275. "
[5] D. S. Taubman and M. W. Mercellin, JPEG 2000: Image Compression Fundamentals, Standards and Practice, Kluwer Academic Publishers, 2001."
[6] G. K. Wallace, The JPEG still picture compression standard, Communications of the ACM, 34 (1991), pp. 30-44."
[7] S. Mallat, A Wavelet Tour of Signal Processing, Academic Press, 1998."
[8] Z. Wang, A.C. Bovik, H.R. Sheikh and E.P. Simoncelli, Image quality assessment: from error visibility to structural similarity, IEEE Transactions on Image Processing , vol.13, no.4 pp. 600- 612, April 2004."https://ece.uwaterloo.ca/~z70wang/research/ssim/index.html"
55 Sampling in image representation and compression