Compressive Sensing
Isidora Stankovic
1 Faculty of Electrical Engineering, University of Montenegro2 GIPSA Lab, INP Grenoble, University of Grenoble Alpes
2nd Conference on Machine Learning for Gravitational Waves,Geophysics and Control Systems
September 2019, Rijeka, Croatia
www.gipsa-lab.grenoble-inp.fr
www.tfsa.ac.me
GIPSA Lab & TFSA Group
Grenoble Image Parole Signal AutomatiquePart of INP Grenoble, CNRS, University of Grenoble Alpes350 people, including 150 permanent staff and 150 PhD studentsInternationally recognized for research in automatic control, signaland image processing, speech and cognitionWeb site: www.gipsa-lab.grenoble-inp.fr
Time Frequency Signal AnalysisPart of Faculty of Electrical Engineering, University of MontenegroGroup of approx. 15 researchers with more than 300 journalpapers, together cited more than 10 000 timesTopics covering: fundamental theory of TFSA, applications such asradar and sonar signal processing, compressive sensing, etc.Web site: www.tfsa.ac.me
I. Stankovic (G2net) Compressive Sensing 11 September, 2019 2 / 35
Presentation outline
1 Introduction
2 Compressive SensingSignal reconstructionGradient reconstruction algorithmApplications
Image processingImpulse noise removalRadar imagingShallow-water acousticsVideo and Ozone density
Effects of nonsparsity
3 Conclusions
I. Stankovic (G2net) Compressive Sensing 11 September, 2019 3 / 35
Introduction
Introduction
A sparse signal is a signal with very few non-zero components incomparison with the total length.Sparse signals can be reconstructed by the theory of compressivesensing with less measurements than with traditionalShannon-Nyquist theorem.Advantages: reduce complexity, improve efficiency (time, storage,etc), can be widely used in many fields.
0 20 40 60 80 100 120−2
0
2
Shannon−Nyquist Sampling
Time
Am
plit
ude
0 20 40 60 80 100 120−2
0
2
Compressive Sensing
Time
Am
plit
ude
I. Stankovic (G2net) Compressive Sensing 11 September, 2019 4 / 35
Compressive Sensing Signal reconstruction
Outline
1 Introduction
2 Compressive SensingSignal reconstructionGradient reconstruction algorithmApplications
Image processingImpulse noise removalRadar imagingShallow-water acousticsVideo and Ozone density
Effects of nonsparsity
3 Conclusions
I. Stankovic (G2net) Compressive Sensing 11 September, 2019 5 / 35
Compressive Sensing Signal reconstruction
Sparse signals
Consider a complex-valued discrete signal x, of length N and itstransform doman denoted by X.
We assume that X is K-sparse vector, K ¿ N. In this case, wecan use M < N samples to reconstruct the whole signal. Thevector with M available samples is denoted by y.
Ideally, measurement y is a linear combination of signalcoefficients with weighting factors ak (such as DFT matrix)
y=N−1∑k=0
ak Xk = a0X0 +a1X1 +a2X2 +·· ·+aN−1XN−1.
In compressive sensing, signal are measured M times (withdifferent weighting coefficients).
I. Stankovic (G2net) Compressive Sensing 11 September, 2019 6 / 35
Compressive Sensing Signal reconstruction
Problem definition
Unavailable samples: desired sampling technique, corruptedsamples, hardaware constraint, etc.The goal of compressive sensing is to minimize the sparsity of Xby knowing a reduced set of the available samples y. Thereconstruction can be formulated as L0-norm optimizationproblem:
min‖X‖0 subject to y=AX
where A is a measurement matrix obtained from the transformmatrix by keeping rows that corresponds to the available samples.Disadvantage: computationally inefficient, sensitive to noise.In reality, general compressive sensing formulation:
min‖X‖1 subject to y=AX
I. Stankovic (G2net) Compressive Sensing 11 September, 2019 7 / 35
Compressive Sensing Signal reconstruction
Reconstruction methods
Norm-zero based reconstructionDirect searchEstimation of the non-zero coefficients positionsMatching pursuit algorithms, OMP, CoSaMP
Norm-one based reconstructionLASSO minimizationL1-magicGradient algorithmTotal variations
Bayesian based reconstruction
I. Stankovic (G2net) Compressive Sensing 11 September, 2019 8 / 35
Compressive Sensing Signal reconstruction
Direct search procedure
In the direct search we try with all possible combinations ofnonzero index values k ∈ {k1,k2, ...,kK }=KThe vector XK contains assumed K nonzero elements of X at thepositions K. The corresponding system with M > K equations issolved in a least square sense
y=AKXK =⇒ XK =(AH
K AK
)−1AH
K y.
For all solutions we check the error y−AKXK . If the error is zerothe reconstruction is successful. If there is more than one result,then the reconstruction is not unique.
The total number of systems that should be solved is(N
K)
→ not feasible.
I. Stankovic (G2net) Compressive Sensing 11 September, 2019 9 / 35
Compressive Sensing Gradient reconstruction algorithm
Outline
1 Introduction
2 Compressive SensingSignal reconstructionGradient reconstruction algorithmApplications
Image processingImpulse noise removalRadar imagingShallow-water acousticsVideo and Ozone density
Effects of nonsparsity
3 Conclusions
I. Stankovic (G2net) Compressive Sensing 11 September, 2019 10 / 35
Compressive Sensing Gradient reconstruction algorithm
Gradient algorithm – basic idea
Assume that we have reduced set of samples (measurements),and that there exist meaningful full set of samples.
We can consider missing samples as variables and try to minimizesparsity measure iteratively by using measure gradient estimation.
Gradient is estimated by using finite difference approach. Eachmissing sample is varied by ±∆ and sparsity measure is calculatedin both cases.
Gradient is proportional to the difference of sparsity measures.
The procedure is repeated for each missing sample. At the end,missing samples are updated according to the calculatedgradients.
Whole procedure is repeated. When gradient vector start tooscillate we can conclude that step ∆ should be reduced.
I. Stankovic (G2net) Compressive Sensing 11 September, 2019 11 / 35
Compressive Sensing Gradient reconstruction algorithm
Algorithm
Input:Set of missing/omitted sample positions QAvailable samples (measurements) yTransformation matrix ΦStep α
Output:Reconstructed signal vector XFull set of measurements x
I. Stankovic (G2net) Compressive Sensing 11 September, 2019 12 / 35
1: m ← 0, Set initial estimate signal vector x(0) as x(0)(ni)= y(i) for ni ∈Mand x(0)(ni)= 0 for ni ∈Q
2: ∆←maxn
|x(0)(n)|3: repeat4: repeat5: x(m+1) ← x(m)
6: for ni ∈Q do . for each missing sample7: z1 ← x(m)
8: z1(ni)← z1(ni)+∆ . try increased value9: z2 ← x(m)
10: z2(ni)← z2(ni)−∆ . try decreased value11: g(ni)←‖Φz1‖1 −‖Φz2‖1 . measure difference
12: x(m+1)(ni)← x(m)(ni)−α g(ni)13: end for14: m ← m+115: until stopping criterion is satisfied . oscillations16: ∆←∆/3 . reduce ∆17: until required precision is achieved18: x← x(m) , X←Φx
Compressive Sensing Gradient reconstruction algorithm
Gradient algorithm – two missing samples
−3
−2
−1
0
1
−3−2
−10
1
2
2.5
3
3.5
4
4.5
5
y(n1)
y(n2)
3.5
3.5
4
4
4
4.5
4.5
4.5
5
55.5
5.5
−4 −3 −2 −1 0 1
−1
−0.5
0
0.5
1
1.5
2
2.5
3
3.5
4
I. Stankovic (G2net) Compressive Sensing 11 September, 2019 14 / 35
Compressive Sensing Gradient reconstruction algorithm
Gradient algorithm – two missing samples
Signal sparse in the DFT domain isconsidered.
The reconstruction error decay rapidlyuntil minimum for a given step ∆ isreached. That should be detectedand we should reduce finite differencestep ∆.
It can be shown that reconstructionerror (precision) is proportional to ∆.
We could use small ∆ from thebeginning, but the number ofiterations will be huge.
0 2 4 6 8
−3
0
3
Available samples
0 20 40 60−80
−60
−40
−20
0
iteration
Reconstruted MSE in [dB]
0 2 4 6 8
−3
0
3
Reconstructed signal in 60 iteratons
time n
I. Stankovic (G2net) Compressive Sensing 11 September, 2019 15 / 35
Compressive Sensing Gradient reconstruction algorithm
Gradient algorithm – example
Consider a signal with N = 64 samples, sparse in the DCT domain withsparsity 3. Assume that only M = 16 samples are available.
I. Stankovic (G2net) Compressive Sensing 11 September, 2019 16 / 35
Compressive Sensing Gradient reconstruction algorithm
Gradient algorithm – example
Reconstructed signal after 100 iterations
I. Stankovic (G2net) Compressive Sensing 11 September, 2019 17 / 35
Compressive Sensing Gradient reconstruction algorithm
Gradient algorithm – reconstruction error
Mean squared error of the reconstruction is presented for each iteration.Iterations when ∆ is reduced are marked with red dots.
0 10 20 30 40 50 60 70 80 90 100
-120
-100
-80
-60
-40
-20
0
I. Stankovic (G2net) Compressive Sensing 11 September, 2019 18 / 35
Compressive Sensing Applications
Outline
1 Introduction
2 Compressive SensingSignal reconstructionGradient reconstruction algorithmApplications
Image processingImpulse noise removalRadar imagingShallow-water acousticsVideo and Ozone density
Effects of nonsparsity
3 Conclusions
I. Stankovic (G2net) Compressive Sensing 11 September, 2019 19 / 35
Compressive Sensing Applications
Image reconstruction
Most of real-world images are sparse in the 2D DCT domain.If some pixels are damaged (missing, corrupted by impulsivenoise, unavailable, bad-pixels in camera CCD sensor,. . . ) we canapply reconstruction algorithm in order to find correct intensities.Consider image with 50% available pixels. Missing (or corrupted)pixels are shown in white color.
I. Stankovic (G2net) Compressive Sensing 11 September, 2019 20 / 35
Compressive Sensing Applications
I. Stankovic (G2net) Compressive Sensing 11 September, 2019 21 / 35
Compressive Sensing Applications
Impulsive noise removal
When the positions of the corrupted pixels in the image are known,the reconstruction is straightforward (salt & pepper noise).Problem: is it possible to detect corrupted pixels when the noise iswithin pixel intensity range ?
Initial iteration: consider all pixels as possibly corrupted.Vary value of each pixel intensity and check the sparsity measure(in the transformation domain) of the obtained images.Pixels with highest changes in sparsity measure are probablycorrupted pixels.Select one or few of them and mark them as corrupted.Reconstruct their values (assuming that other pixels are correct)and continue search in the same manner.
I. Stankovic (G2net) Compressive Sensing 11 September, 2019 22 / 35
Compressive Sensing Applications
Image denoising – results
50% pixels are corrupted with combined noise(salt & pepper and Gaussian)
Noisy image Reconstructed image
I. Stankovic (G2net) Compressive Sensing 11 September, 2019 23 / 35
Compressive Sensing Applications
ISAR imaging – classical full data approach
signal after coherent processing
received signal
transmitted signal – series of M pulses
2D signal
ISAR image
2D FT
coherent integration time
pulse width
I. Stankovic (G2net) Compressive Sensing 11 September, 2019 24 / 35
Compressive Sensing Applications
ISAR imaging – reduced number of pulses
signal after coherent processing
received signal
transmitted signal
2D signal
ISAR image
2D FT
?
coherent integration time
pulse width
× × ×
× × ×
× × ×
×
×
×
I. Stankovic (G2net) Compressive Sensing 11 September, 2019 25 / 35
Compressive Sensing Applications
ISAR imaging – MIG example
50% pulses are omitted.The reconstruction is performed by gradient algorithm.
cross−range
range
100% available pulses
cross−range
range
30% available pulses
cross−range
range
30% − reconstructed image
cross−range
range
50% available pulses
cross−range
range
50% − reconstructed image
cross−range
range
70% available pulses
cross−range
range
70% − reconstructed image
I. Stankovic (G2net) Compressive Sensing 11 September, 2019 26 / 35
Compressive Sensing Applications
Shallow-water acoustics
Transmitted - linearly frequency modulated signal, u(n)= e jπαn2.
Propagates through anunderwater dispersivechannel
The transfer function of thereceived signal:
H( f )≈+∞∑m=1
A(m, f )pr
exp{ jkr(m, f )r}.1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2
100
200
300
400
500
600
Our goal - decompose the mode functions, which will make theproblem of detecting the transmitted signal straightforward.
I. Stankovic (G2net) Compressive Sensing 11 September, 2019 27 / 35
Compressive Sensing Applications
Video and Ozone density
Figure: captionI. Stankovic (G2net) Compressive Sensing 11 September, 2019 28 / 35
Compressive Sensing Effects of nonsparsity
Outline
1 Introduction
2 Compressive SensingSignal reconstructionGradient reconstruction algorithmApplications
Image processingImpulse noise removalRadar imagingShallow-water acousticsVideo and Ozone density
Effects of nonsparsity
3 Conclusions
I. Stankovic (G2net) Compressive Sensing 11 September, 2019 29 / 35
Compressive Sensing Effects of nonsparsity
Influence of missing samples
The initial STFT is calculated using the available samples only
SN0(n,k)=M∑
i=1x(n+mi)w(mi)e− j 2π
N mik
0 20 40 60 80 100 120
-2
0
2
0 20 40 60 80 100 120
0
20
40
60
80
0 20 40 60 80 100 120
-2
0
2
0 20 40 60 80 100 120
0
20
40
60
80
0 20 40 60 80 100 120
-2
0
2
0 20 40 60 80 100 120
0
20
40
60
80
I. Stankovic (G2net) Compressive Sensing 11 September, 2019 30 / 35
Compressive Sensing Effects of nonsparsity
Nonsparse signal reconstruction
Energy in the reconstructed signal is proportional to the energy ofnonreconstructed components (sum of squared amplitudes)
‖SNR(n)−SNK (n)‖22 = K
NM
MN‖SN(n)−SNK (n)‖2
2 .
50 100 150 200 250
0
50
100
150
200
250
300
350
I. Stankovic (G2net) Compressive Sensing 11 September, 2019 31 / 35
Compressive Sensing Effects of nonsparsity
Example
Two linear frequency modulated signal components
Hamming window N = 256, M = 192 available samples
Reconstructed using orthogonal matching pursuit algorithm
0 100 2000
16
32
0
0.2
0.4
time
frequency 0 100 2000
16
32
0
0.2
0.4
time
frequency
0 100 2000
16
32
0
0.2
0.4
time
frequency 0 100 2000
16
32
0
0.2
0.4
time
frequency
Figure: Original (top left); K=8 (top right); K=16 (bottom left); K=32(bottom right)
I. Stankovic (G2net) Compressive Sensing 11 September, 2019 32 / 35
Compressive Sensing Effects of nonsparsity
Example - train
Assumed sparsity K = 55 and 50% available samples
Original STFT
10 20 30 40 50
100
200
300
400
500
Available STFT samples
10 20 30 40 50
100
200
300
400
500
Reconstructed STFT
10 20 30 40 50
100
200
300
400
5000 10 20 30 40 50 60
−20
−15
−10
−5
0
5Total error in signal
To
tal
erro
r in
dB
Assumed sparsity K
I. Stankovic (G2net) Compressive Sensing 11 September, 2019 33 / 35
Conclusions
Summary and conclusions
Acquisition technique = relatable to many fields.Robust algorithms for successful reconstruction of unavailablemeasurements.Possible tool for signal/image denoising.Future in gravitational waves?
R. Inta, “Sparse Methods for Gravitational Wave Detection,” The Australian National University
I. Stankovic (G2net) Compressive Sensing 11 September, 2019 34 / 35
Conclusions
Compressive Sensing
Thank you for your attention.
Questions?
I. Stankovic (G2net) Compressive Sensing 11 September, 2019 35 / 35