Single-letter Characterization of Signal Estimation
from Linear Measurements
Dongning Guo
Dror Baron
Shlomo Shamai
The work has been supported by the European Commission in the framework of the FP7 Network of Excellence in Wireless Communications NEWCOM++, by the Israel Science Foundation, and by the National Science Foundation.
Linear Measurement Systems
• 1809: Theoria motus corporum coelestium • Gauss introduced application of least squares
(regression) to solve noisy linear systems – motivated by astronomy/navigation
• Goal: estimate input x to explain measurements y
Non-linear Signal Estimation • Linear signal estimation (least squares) sub-optimal
– example: “hard decisions” used to estimate binary data
• Difficult problem with noisy observations – even over-determined problems can be challenging
Need information theoretic framework for non-linear signal estimation in linear measurement systems
underdetermined overerdetermined
Linear Measurement Application Areas
Compressed sensing
Multiuser communication (CDMA)
• Medical imaging (tomography)
• Financial prediction
• Electromagnetic scattering
• Seismic imaging (oil industry)
Problem Definition
Setting • Replace samples by more general measurements
based on a few linear projections (inner products)
measurements sparse signal
# non-zeros
Signal Model • Signal entry Xn= BnUn
• iid Bn» Bernoulli(ε) sparse • iid Un» PU
PU
Bernoulli(ε)
Multiplier
PX
Non-Sparse Input
• Can use ε=1 Xn= Un
PU
Measurement Noise • Measurement process is typically analog • Analog systems add noise, non-linearities, etc.
• Assume Gaussian noise for ease of analysis
• Can be generalized to non-Gaussian noise [Guo & Wang 2007; Rangan 2010]
• Noiseless measurements denoted y0 • Noise • Noisy measurements • Unit-norm columns SNR=γ
Noise Model
noiseless
SNR
• Model process as measurement channel
• Measurements provide information!
Preliminary single-letter bound for compressed sensing and linear measurement systems
channel
CS measurement CS decoding
source encoder
channel encoder
channel decoder
source decoder
Allerton 2006 [Sarvotham, Baron, & Baraniuk]
• Numerous single-letter bounds – [Aeron, Zhao, & Saligrama] – [Akcakaya and Tarokh] – [Rangan, Fletcher, & Goyal] – [Gastpar & Reeves] – [Wang, Wainwright, & Ramchandran] – [Tune, Bhaskaran, & Hanly]
• BP Multiuser detection – [Tanaka & Takeda] – [Guo & Wang] – [Montanari & Tse]
• Arbitrary noise – [Rangan] – [Guo & Wang]
Related Results
Goal: Precise Single-letter Characterization of Optimal CS
[Guo, Baron, & Shamai 2009]
What Single-letter Characterization?
• Ultimately what can one say about Xn given Y?
(sufficient statistic)
• Very complicated • Want a simple characterization of its quality • Large-system limit:
Φ ,Φ
channel posterior
Main Result: Single-letter Characterization
• Result1: Conditioned on Xn=xn, the observations (Y,Φ) are statistically equivalent to
η easy to compute…
• Estimation quality from (Y,Φ) just as good as noisier scalar observation
degradation
Φ ,Φ
channel posterior
• η2(0,1) is fixed point of
• Take-home point: degraded scalar channel
• Non-rigorous owing to replica method w/ symmetry assumption – used in CDMA detection [Tanaka 2002, Guo & Verdu 2005]
• Related analysis [Rangan, Fletcher, & Goyal 2009] – MMSE estimate (not posterior) using [Guo & Verdu 2005]
– extended to several CS algorithms particularly LASSO
Details
Decoupling [Guo, Baron, & Shamai 2009]
• Result2: Large system limit; any arbitrary (constant) L input elements decouple:
• Take-home point: “interference” from each individual signal entry vanishes
Decoupling Result
Sparse Measurement Matrices
[Sarvotham, Baron, & Baraniuk 2006] [Guo, Baron, & Shamai 2009]
[Baron, Sarvotham, & Baraniuk 2010]
Sparse Measurement Matrices
• LDPC measurement matrix (sparse) • Mostly zeros in Φ; nonzeros » PΦ
• Each row contains ¼Nq randomly placed nonzeros • Fast matrix-vector multiplication
fast encoding / decoding
sparse matrix
CS Decoding Using BP [Baron, Sarvotham, & Baraniuk 2006]
• Measurement matrix represented by graph • Estimate input iteratively • Implemented via nonparametric BP [Bickson,Sommer,…]
measurements y
signal x
Identical Single-letter Characterization w/BP [Montanari & Tse 2006; Guo & Wang 2008]
• Result3: Conditioned on Xn=xn, the observations (Y,Φ) are statistically equivalent to
• Rigorous result
• Sparse matrices just as good • BP is asymptotically optimal!
identical degradation
100 200 300 400 500 600 700
20
30
40
50
60
70
80
90
100
IHT
CoSaMP
GPSR
LP
M
MMSE
CS-BP vs Other CS Methods (N=1000, ε=0.1, q=0.02)
M
MM
SE
Conclusion
• Single-letter characterization of CS
• Decoupling
• Sparse matrices just as good
• Asymptotically optimal CS-BP algorithm
THE END
