Hybrid Dense/Sparse Matrices in Compressed Sensing Reconstruction

Ilya Poltorak

Dror Baron

Deanna Needell

The work has been supported by the Israel Science Foundation and National Science Foundation.

CS Measurement• Replace samples by more general encoder

based on a few linear projections (inner products)

measurements sparsesignal

# non-zeros

Caveats

• Input x strictly sparse w/ real values• Noiseless measurements

– noise can be addressed (later)

• Assumptions relevant to content distribution (later)

Why is Decoding Expensive?

measurementssparsesignal

nonzeroentries

Culprit: dense, unstructured

Sparse Measurement Matrices (dense later!)

measurementssparsesignal

nonzeroentries

• LDPC measurement matrix (sparse)• Only {-1,0,+1} in • Each row of contains L randomly placed nonzeros • Fast matrix-vector multiplication

fast encoding & decoding

Example

0

1

1

4

0 1 1 0 0 0

0 0 0 1 1 0

1 1 0 0 1 0

0 0 0 0 1 1

?

?

?

?

?

?

Example

0

1

1

4

0 1 1 0 0 0

0 0 0 1 1 0

1 1 0 0 1 0

0 0 0 0 1 1

?

?

?

?

?

?

• What does zero measurement imply?• Hint: x strictly sparse

Example

0

1

1

4

0 1 1 0 0 0

0 0 0 1 1 0

1 1 0 0 1 0

0 0 0 0 1 1

?

0

0

?

?

?

• Graph reduction!

Example

0

1

1

4

0 1 1 0 0 0

0 0 0 1 1 0

1 1 0 0 1 0

0 0 0 0 1 1

?

0

0

?

?

?

• What do matching measurements imply?• Hint: non-zeros in x are real numbers

Example

0

1

1

4

0 1 1 0 0 0

0 0 0 1 1 0

1 1 0 0 1 0

0 0 0 0 1 1

0

0

0

0

1

?

• What is the last entry of x?

Noiseless Algorithm [Luby & Mitzenmacher 2005]

[Sarvotham, Baron, & Baraniuk 2006][Zhang & Pfister 2008]

Phase1: zero measurements

Phase2: matching measurements

Phase3: singleton measurements

Initialize

Done? Arrange outputyesno

typically iterate 2-3 times

Numbers (4 seconds)

• N=40,000• 5% non-zeros• M=0.22N• L=20 ones per row

Only 2-3 iterations

Iteration, Phase

Zeros Non-zeros Total

1,1 30615 0 30615

1,2 35224 977 36201

1,3 35224 1500 36724

2,1 36800 1500 38300

2,2 37180 1833 39013

2,3 37180 2063 39243

3,1 37268 2063 39331

3,2 37289 2074 39363

3,3 37289 2083 39372

4,1 37289 2083 39372

4,2 37291 2084 39375

4,3 37291 2084 39375

iteration #1

Challenge

• With measurements parts of signal still not reconstructed

• How do we recover the rest of the signal?

Solution: Hybrid Dense/Sparse Matrix

• With measurements parts of signal still not reconstructed

• Add extra dense measurements • Residual of signal w/ residual dense columns

residual columns

Sudocodes with Two-Part Decoding [Sarvotham, Baron, & Baraniuk 2006]

• Sudocodes (related to sudoku)• Graph reduction solves most of CS

problem

• Residual solved via matrix inversion

sudo decoder

Residual via matrix inversion

residual columns

Contribution 1: Two-Part Reconstruction• Many CS algorithms for sparse matrices

[Gilbert et al., Berinde & Indyk, Sarvotham et al.]

• Many CS algorithms for dense matrices[Cormode & Muthukrishnan, Candes et al., Donoho et al., Gilbert et al., Milenkovic et al., Berinde & Indyk, Zhang & Pfister, Hale et al.,…]

• Solve each part with appropriate algorithm

residual columns

sparse solver

residual via dense solver

Runtimes (K=0.05N, M=0.22N)

Theoretical Results [Sarvotham, Baron, & Baraniuk 2006]

• Fast encoder and decoder– sub-linear decoding complexity– caveat: constructing data structure

• Distributed content distribution– sparsified data– measurements stored on different servers– any M measurements suffice

• Strictly sparse signals, noiseless measurements

Contribution 2: Noisy Measurements

• Results can be extended to noisy measurements

• Part 1 (zero measurements): measurement |ym|<

• Part 2 (matching): |yi-yj|<

• Part 3 (singleton): unchanged

Problems with Noisy Measurements

• Multiple iterations alias noise into next iteration!Use one iteration

• Requires small threshold (large SNR)

• Contribution 3: Provable reconstruction• deterministic & random variants

Summary

• Hybrid Dense/Sparse Matrix– Two-part reconstruction

• Simple (cute?) algorithm

• Fast

• Applicable to content distribution

• Expandable to measurement noise

THE END