WAVELETS: Data Analytic Perspectivezoe.bme.gatech.edu/~bv20/isye6414/Bank/SNUtalk1.pdf · WAVELETS:...

transcript

WAVELETS: Data Analytic Perspective

Brani Vidakovic

Georgia Institute of Technology, Atlanta, GA, USA

Seminar Talk at Department of Industrial EngineeringSeoul National University

Vidakovic, B. (GaTech) WAVELETS: Data Analytic View July 11, 2013 1 / 44

Overview

In this talk:

Overview

In this talk:Four Holy Grails of Wavelets or Why Wavelets

Overview

What are Wavelets?

Overview

What are Wavelets?

Wavelet Shrinkage via Statistical Inference

Overview

What are Wavelets?

BAMS Example

Overview

What are Wavelets?

BAMS Example

Wavelets and Scaling

Overview

What are Wavelets?

BAMS Example

MATLAB Sessions

Overview

What are Wavelets?

BAMS Example

MATLAB Sessions

MATLAB DEMOS:http://zoe.bme.gatech.edu/~bv20/isye6420/

Overview

What are Wavelets?

BAMS Example

MATLAB Sessions

MATLAB DEMOS:http://zoe.bme.gatech.edu/~bv20/isye6420/

Time/Frequency or Time/Scale Domains

Echolocation Signal in Time/Scale Domain

0 50 100 150 200 250 300 350 400−0.25

−0.2

−0.15

−0.1

−0.05

0 50 100 150 200 250 300 350 4000

Figure: Digitized 2.5 microsecond echolocation pulse emitted by theLarge Brown Bat, Eptesicus Fuscus and time/scale (Wigner-Ville)representation of the pulse. (Left) Echolocation principle; (Middle)Pulse in the time domain; (Right) Wigner-Ville transform.Data courtesy of Rich Baraniuk, DSP at Rice University.

Wavelets compress the data

0 100 200 300 400 500

elet d

0 100 200 300 400 500

0.0 0.2 0.4 0.6 0.8 1.0

Figure: Left: Normalized wind velocity [60 Hz, Duke Forest] timeseries and its wavelet transform. Right. Corresponding Lorentz curvesof “energies” (squared coefficients).

Wavelets whiten the data

n0 5 10 15 20

0 5 10 15 20

Figure: The autocorrelations in the wind velocity time series [panel(a)] and its wavelet transform [panel (b)].

Wavelets filter the data

0 1 2 3 4 5 6

−0.8

−0.6

−0.4

−0.2

0 1 2 3 4 5 6

−0.5

2 3 4 5 6 7 8 90

0 1 2 3 4 5 6−1

−0.8

−0.6

−0.4

−0.2

Figure: Two functions with different frequencies/scale informationseparate in the wavelet domain.

Wavelets detect self-similarity in data

1 2 3 4 5 6

−0.8

−0.6

−0.4

−0.2

4 6 8 10 12 14

−1.9839

dyadic level

log2(average(coefs2))

Figure: (Left) A path of Brownian motion; (Right) Wavelet basedlog-spectrum. Regular decay is a signature of monofractality.

What are Wavelets?

• IL2 o.n. bases of the form

ψjk(x) = 2j/2ψ(2jx− k), j, k ∈ ZZ}

f ∈ IL2 : f(x) ≈∑

j,k ∈ finite set

djkψjk(x).

j - index of scale with resolution/size 2−j. Frequency is 2j , areciprocal of scale.k - location, shift, translate, “time”. Measures the energy in theneighborhood of x = k/2j.

• {φJ0,k(x), ψjk(x), j ≥ J0, k ∈ ZZ}• Multiresolution Analysis (MRA) fully determined by φ

Three Ways To Think About DWT

y˜= (y1, . . . , yN)- data. There is an underlying function f , so

thatf(x) =

ykφJk(x), J = log2(N)

1. f(x) =∑

j<J,k djkψjk(x) y˜−→ {djk = 〈f, ψjk〉} or d

thatf(x) =

1. f(x) =∑

2. d˜=Wy

˜or y

˜= W ′d

thatf(x) =

1. f(x) =∑

2. d˜=Wy

˜or y

˜= W ′d

3. d˜= (...G(H(H(Hy

˜))),G(H(Hy

˜)),G(Hy

˜),Gy

Second & Third!

2.d˜=Wy

˜(Short signals and images)

h˜= (h1, . . . , hM) g

˜via gn = (−1)n h1−n.

Haar: h˜= (1/

√2 1/

√2) g

˜= (1/

√2 − 1/

H - filtering with h˜(low pass) + decimation (keep every 2nd)

G - filtering with g˜(high pass) + decimation

d˜= (H . . . (H(H(Hy

˜))) | G . . . (H(H(Hy

˜))) | . . . | G(H(Hy

˜)) | G(Hy

˜) | Gy

d˜= (smooth part | coarsest details | . . . | finest details )

Organization of Scales in Wavelet Domain

signal y˜

1024finest details Gy

fine details G(H(y˜)) 256

details G(H(H(y˜))) 128

coarse details G(H(H(H(y˜)))) 64

coarsest details G(H(H(H(H(y˜))))) 32

smooth H(H(H(H(H(y˜))))) 32

Mallat’s Algorithm

✲ ✲ ✲ ✲ ✲y˜

c˜(1) c

˜(2) . . . c

˜(k−1) c

✡✡✡✡✡✣

d˜(1) d

˜(2) . . . d

˜(k−1) d

d˜(3)

c˜(3)

. . .⊕

❏❏❏❏❫

✲c˜(2)

d˜(2)

❏❏❏❏❫⊕✲ ✲c

d˜(1)

❏❏❏❏❫⊕✲ ✲ y

Mallat’s Algorithm

H,G a single step in forward DWT filter + ↓H∗,G∗ a single step in inverse DWT ↑ + filter

H : cj−1,l =∑

hk−2lcj,k

G : dj−1,l =∑

gk−2lcj,k

H∗,G∗ : cj,k =∑

cj−1,lhk−2l +∑

dj−1,lgk−2l.

dwtr.m: wdata = dwtr(data, L, filterh)idwtr.m: data1 = dwtr(wdata, L, filterh)

DEMO 0

• Dem0a.m (dwtr.m, idwtr.m)• Dem0b.m (WavMat.m)

200 400 600 800 1000−0.5

0.5Doppler

200 400 600 800 1000−2

2Wavelet Transform of Doppler Signal by Symmlet 8

200 400 600 800 1000−0.5

0.5Doppler Recovered

Figure: Output of Dem0a.m

From h to φ, ψ

h → m0 → Φ(ω) → φ(x)

Transfer function and Φ(ω)

m0(ω) =1√2

k∈ZZhke

−ikω [=1√2H(ω)].

Φ(ω) = F(φ(x)) =∫ ∞

−∞φ(x)e−iωxdx

φ(x) =∑

k hk√2φ(2x− k)

Φ(ω) = m0

= · · · = ∏∞n=1m0

. φ(x) = F−1(Φ(ω))

DEMO 1

• Dem1a.m (Symmlet 4, via Daubechies-Lagarias Algorithm, uses Phijk.m,Psijk.m)• h = [-0.07576571479 -0.02963552765 0.49761866763 0.803738751810.29785779561 -0.09921954358 -0.01260396726 0.03222310060];

• Dem1b.m (Pollen family: h = [(1 + cosφ− sinφ)/s,

(1 + cosφ+ sinφ)/s, (1− cosφ+ sinφ)/s, (1− cosφ− sinφ)/s] for s = 2√2

and φ = π/4)

0 2 4 6

−2 0 2 4

−0.6

−0.4

−0.2

0 1 2 3

−0.5

−1 0 1 2

−0.5

Figure: Scaling and wavelet functions for Symmlet 4 and Pollen ϕ = 45◦

Why Wavelets in Data Analysis?

WT are local (in time; in scale/frequency)

WT are orthogonal or near-orthogonal

WT are applicable to discrete data sets

WT preserve but disbalance the energy in data

WT whiten data

WT are FAST! O(n)

WT whiten data

WT are FAST! O(n)

WT are versatile

WT whiten data

WT are FAST! O(n)

WT are versatile

WT are Bayes friendly

WT whiten data

WT are FAST! O(n)

WT are versatile

WT are sensitive to self-similar phenomena

WT whiten data

WT are FAST! O(n)

WT are versatile

WT are sensitive to self-similar phenomena

much more...

Where Wavelets in Data Analysis?

• Regression Problems♦ Equi- and Nonequi-spaced Regression♦ Dimension Reduction, Approximate PCA, Denoising♦ Pursuits and Data Driven Selection of Analyzing Tools

• Density Estimation♦ Functionals of a Density, Classification and Discrimination.

• Deconvolutions

• Time Series♦ Approximate K-L Expansions.♦ Nonstationary TS, Wavelet-based Spectral Analysis

• Deconvolutions

• Time Series♦ Approximate K-L Expansions.♦ Nonstationary TS, Wavelet-based Spectral Analysis

• Long-Range Dependence, Self-similarity and Scaling in Data,(Multi-)Fractality.

Where Wavelets in Statistics?

• GLM, GAM, Censored Models, Functional Data Analysis,Change Point Analysis

• Theory of Shapes, Wavelet Based Bookmarks

• Biased Sampling

• Medical Image Enhancement, Mammogramy, fMRI, CXR, CT

• Biased Sampling

• Bayesian Applications. Bayesian Nonparametrics

• Biased Sampling

• Statistical Calculation, Simulation, Wavestrapping.

• Biased Sampling

• Statistical Calculation, Simulation, Wavestrapping.

Model Based Wavelet Data Processing

DATAW−→ Wavelet Coefficients: “Detail” & “Smooth”

Processed DATAW−1

←− Process (Detail) Coefficients

Process ≡

Processed DATAW−1

Process ≡Shrink, Threshold, Split

Processed DATAW−1

Transform

Processed DATAW−1

TransformSimulate New, Construct

Processed DATAW−1

Resample, Permute

Processed DATAW−1

Resample, PermuteAssess “Energy” and “Fluxes”

Wavelet Shrinkage: Shrinkage Policies

Hard ThresholdingSoft ThresholdingSemi-Soft ThresholdingSmooth ShrinkageVarious Variable Selection Methods

-1 -0.5 0.5 1d

-4 -2 2 4

Figure: Examples of thresholding rules

Universal Threshold

λ =√

2 logn σ̂

σ̂ is an estimator of std of noise, σ. Many proposals – usuallyinvolving only the finest level of detail.

Rationale: If X1, . . . , Xn are i.i.d. N (0, σ2) thenEX(n) = −EX(1) =

√2 logn σ.

• Since the estimated expected range of noise is[ −√2 logn σ̂, √2 logn σ̂ ], any coefficient with the magnitudeoutside the range is attributed to the signal and thus retained inthe model.• Oversmooths in practice.

DEMO 2

• Dem2a.m, Dem2b.m (Universal Threshold, HardThresholding Policy)

0 0.2 0.4 0.6 0.8 1−0.8

−0.6

−0.4

−0.2

0 0.2 0.4 0.6 0.8 1−0.8

−0.6

−0.4

−0.2

Figure: Left: Doppler + Noise (data in red) and Doppler(green). Right: Doppler estimate by thresholding (black).

Shrinkage induced by statistical modeling in thewavelet domain

y˜= f

˜+ ǫ˜

W−→ d˜= θ

˜+ ǫ˜

Estimate θ by θ̂ and obtain f̂ as W−1(θ̂)

Location model on d, f(d− θ˜|parameters)

Dimensionality (Do not worry – wavelets decorrelate)Accounting for dependence (neighbors, parent-children),

Blocking strategies (classical), Many Bayes solutions (MCMC,hidden MC’s).

Model complexity/efficiency compromise. Simplemodels/Fast shrinkage ◦ Realistic? Complex models ◦Useful?

Compromise between model reality and simplicity

BAMS (Bayesian Adaptive MultiscaleShrinkage/Smoothing)

Model (Likelihood)

[d|θ, σ2] ∼ N (θ, σ2); σ2 ∼ E(µ), µ > 0;[

µe−µσ2

, µ > 0]

Marginal Likelihood

d|θ ∼ DE(

θ,1√2µ

2µe−√2µ|d−θ|

[θ|ǫ] ∼ ǫδ0 + (1− ǫ)DE(0, τ), ǫ = ǫ(multiresolution level)

Predictive Distribution – Marginal

d ∼ m(d) = ǫDE(0, 1√2µ

) + (1− ǫ)τe−|d|/τ − 1√

2µe−

√2µ|d|

2τ 2 − 1/µ

−3 −2 −1 0 1 2 30

Bayes Rule: BAMS

δ(d) = (1− ǫ)m(d)δ∗(d)/

(1− ǫ)m(d) + ǫDE(0, 1√2µ

δ∗(d) =τ(τ2 − 1/(2µ))de−|d|/τ + τ2/µ(e−|d|√2µ − e−|d|/τ)

(τ2 − 1/(2µ))(τe−|d|/τ − (1/√2µ)e−|d|√2µ)

−6 −4 −2 0 2 4 6

Figure: Bayes’ rule for selected values of τ , µ and ǫ.Vidakovic, B. (GaTech) WAVELETS: Data Analytic View July 11, 2013 27 / 44

Specification of Hyperparameters:• Recall: σ2 ∼ E(µ), IEσ2 = 1/µ ⇒ µ = 1

pseudos ,

(Tukey) pseudos = |Q1 −Q3|/C where Q1 and Q3 are the firstand the third quartile of the finest level of details in thedecomposition and 1.3 ≤ C ≤ 1.5.• ǫ(j) = 1− 1

(j−coarsest+1)γ, γ = 3

• τ = max{√

σ2d − 1

µ, 0}. (Information on selfsimilarity via τ)

0 0.2 0.4 0.6 0.8 1−15

Figure: Doppler signal: n = 1024, SNR=7; Noisy version (left) andVidakovic, B. (GaTech) WAVELETS: Data Analytic View July 11, 2013 28 / 44

DEMO 3

Dem3a.m (BAMS shrinkage, Enrico Caruso: E lucevan lestelle from Tosca, by G. Puccini, Recorded in February1904).

0 1 2 3 4 5

−5000

Figure: Noisy Recording; “Denoised” Sound; Residuals

Scaling: It Started with Hurst and Nile Data

Harold Edwin Hurst was a poor Leicester boy who madegood, eventually working his way into Oxford, and later became aBritish “civil servant” in Cairo in 1906. He got interested in theNile River.

Hurst spent 62 years in Egypt mostly working on designand construction of reservoirs along the Nile.

By inspecting historical data on the Nile flows, Hurstdiscovered phenomenon (now called Hurst effect).

Hurst’s Problem

Optimal reservoir capacity R to accept the river flow in N unitsof time, X1, X2, . . .XN , with a constant withdrawal of X̄ per unittime. The optimal volume of the reservoir is adjusted range,

R = max1≤k≤N

(X1 + · · ·+Xk − kX̄)− min1≤k≤N

(X1 + · · ·+Xk − kX̄).

In order to compare 100 of years worth of data, Hurststandardized the adjusted ranges R, with sample standarddeviation

N − 1

(Xi − X̄)2 ,

Dimensionless ratio R/S - rescaled adjusted range.Vidakovic, B. (GaTech) WAVELETS: Data Analytic View July 11, 2013 31 / 44

River Nile and Scaling

On basis of more that 800 records, Hurst found thatquantity R/S scales as NH , for H ranging from 0.46 to 0.93, withmean 0.73 and standard deviation of 0.09.

0.4 0.5 0.6 0.7 0.8 0.9 10

Easy: H = 1/2 for iid normal; Feller: any iid with finite variance;Barnard (1956) Markovian dependent variables.

Mandelbrot (1975), Mandelbrot and Van Ness (1968), andMandelbrot and Wallis (1968) associated the Hurst phenomenonwith the presence of long-memory (

iCov(X1, Xi) =∞).Vidakovic, B. (GaTech) WAVELETS: Data Analytic View July 11, 2013 32 / 44

River Nile and Scaling

0 100 200 300 400 5009

0 1 2 3 4 5 6 7 8 9−3

slope=−0.80

Dyadic Scale

Figure: (a) Nile Yearly Minimum Level Data for n = 512 ConsecutiveYears from 622 A.D.; (b) Wavelet Log-spectrum[0.80 = 2H − 1→ H = 0.90]

• But what is Wavelet Log-spectrum?Vidakovic, B. (GaTech) WAVELETS: Data Analytic View July 11, 2013 33 / 44

What is Wavelet (Log)Spectrum?

• y - data, n× 1, n = 2J .• d =Wy - discrete wavelet transform, n× 1, n = 2J .• d={cJ−m, dJ−m, dJ−m+1, . . . , dJ−2, dJ−1}.

•Wavelet(Log)Spectrum

j, log2[12j

k d2j,k]

, J −m ≤ j ≤ J − 1}

• SLOPE either −(2H + 1)(cumulative) or −(2H − 1)(differenced). For example, Brownian Motion and White Noiseboth share H = 0.5.• MATLAB’s function waveletspectra.m finds and plots waveletspectra.

“Ubiquitous” – The Epithet of Scaling

Atmospheric Turbulence

0 1 2 3 4 5 6

locity

0 1 2 3 4 5 6 7 8−20

slope = −5/3

0 5 10 15−15

slope = − 5/3

Dyadic Scales

Figure: (a) U Velocity Component; (b) Scaling in the Fourier Domain;(c) Scaling in the Wavelet Domain. [5/3 = 2H + 1→ H = 1/3]

DNA Scales

A DNA molecule consists of long complementary double helix ofpurine nucleotides (denoted as A and G) and pyrimidinenucleotides (denoted as C and T). [A,G→ +1; C, T → −1]

0 1000 2000 3000 4000 5000 6000 7000 8000

3 4 5 6 7 8 9 10 11 12−15

slope=−2.24

Figure: (a) 8196-long DNA Walk for Spider Monkey, from EMLBNucleotide Sequence Alignment DNA Database; (b) Wavelet ScalingWith Slope −2.24. [2.24 = 2H + 1→ H = 0.62]

Money scales

Daily rates of exchange between Korean Won (₩) and Euro (€)as reported by the European Central Bank between 4 January1999 and 21 June 2013, http://sdw.ecb.europa.eu

0 500 1000 1500 2000 2500 3000 3500 4000800

0 2 4 6 8 10

−1.99392

dyadic level

2048 days starting with 1/4/1999

0 2 4 6 8 10

−1.99413

dyadic level

2048 days prior to 6/21/2013

Figure: (a) Daily exchange Rates of ₩ to € between 1/4/1999 and6/21/2013 (Source European Central Bank) (b) Scaling behavior in the“red interval” and (c) in the “green” interval. [2 ≈ 2H + 1→ H = 1/2]

Various measurements scale!

Other Examples

Various Geophysical High Frequency Measurements.Biometric ResponsesEconomic IndicesInternet Measurements.Industrial Measurements.Astronomy.Medicine. Brain and Cancer Research

DEMO 4

• Dem4a.m (Fractional Brownian Motion)• Dem4b.m (DNA RW)• Dem4c.m (Scaling of Exchange Rates ₩ vs €)• Dem4d.m (Gait Data)• Dem4e.m (Coca Cola Company Stocks)

Dem4b.m Output

1000 2000 3000 4000 5000 6000 7000 8000

−400

−350

−300

−250

−200

−150

−100

0 2 4 6 8 10 12

−2.24122

dyadic level

DEMO 4

Demo4d.m Output

500 1000 1500 2000

−0.04

−0.02

500 1000 1500 2000

1 2 3 4 5 6 7 8 9 10

−7−1.02486

dyadic level

1 2 3 4 5 6 7 8 9 10

−2.88611

dyadic level

Figure: Gait Data: Timing between steps. Slope for the cumulativetime is −(2H + 1) = −2.81 → H = 0.905.

Case Study: Filtering Ambient Ozone

Katul, Ruggeri, and Vidakovic (2005), JSPI

0 5 10 15 20 2510

time (min)

1 2 3 4 5 6 7 8 9 10−6

log scalelo

Figure: Raw data from a Gas-Analyzer (21Hz); Wavelet Spectra.

Ozone Case Study

Estimator of O3

1 2 3 4 5 6 7 8 9 10−15

log scale

0 5 10 15 20 2510

time (min)e

Figure: De-whitened Spectra; Estimator of O3 Concentration.

Ozone Case Study

0 5 10 15 20 25

time (min)

−30 −20 −10 0 10 20 3010

0 5 10 15 20 25 30−0.4

−0.2

Conclusions

Wavelets becoming standard tools (like Fourier transform)

Conclusions

Wavelet shrinkage and scaling assessment: Useful tools indata analysis.

Conclusions

Goal of the talk was to build intuition about wavelet dataprocessing and demonstrate fundamental operational concepts.

Conclusions

Goal of the talk was to build intuition about wavelet dataprocessing and demonstrate fundamental operational concepts.

MATLAB DEMOS:http://zoe.bme.gatech.edu/~bv20/isye6420/supporting.html;(Under April 14, 2015 entry).

WAVELETS: Data Analytic Perspectivezoe.bme.gatech.edu/~bv20/isye6414/Bank/SNUtalk1.pdf · WAVELETS:...

Documents