Introduction Estimators Derivations Wavelet use

Sparsity 101: Statistical estimatorsCentral location and dispersion

Laurent Duval

IFP Energies nouvelles

2013

Laurent Duval: Sparsity 101: Statistical estimators Central location and dispersion IFP Energies nouvelles

Introduction Estimators Derivations Wavelet use

2/7

IntroductionWhat is the trend? Where is the outlier?

0 2 4 6 8 100

2

4

6

8

10

12

14

16

18

20

Figure : Toy noise problem

Laurent Duval: Sparsity 101: Statistical estimators Central location and dispersion IFP Energies nouvelles

Introduction Estimators Derivations Wavelet use

2/7

IntroductionWhat is the trend? Where is the outlier?

0 2 4 6 8 100

2

4

6

8

10

12

14

16

18

20

Figure : Toy noise problem

Laurent Duval: Sparsity 101: Statistical estimators Central location and dispersion IFP Energies nouvelles

Introduction Estimators Derivations Wavelet use

2/7

IntroductionWhat is the trend? Where is the outlier?

0 2 4 6 8 10

−2

0

2

4

6

8

10

12

14

16

18

20

Figure : Toy noise problem

Laurent Duval: Sparsity 101: Statistical estimators Central location and dispersion IFP Energies nouvelles

Introduction Estimators Derivations Wavelet use

2/7

IntroductionWhat is the trend? Where is the outlier?

0 2 4 6 8 10

−2

0

2

4

6

8

10

12

14

16

18

20

Figure : Toy noise problem

Laurent Duval: Sparsity 101: Statistical estimators Central location and dispersion IFP Energies nouvelles

Introduction Estimators Derivations Wavelet use

2/7

IntroductionWhat is the trend? Where is the outlier?

0 2 4 6 8 10

−2

0

2

4

6

8

10

12

14

16

18

20

0.11111

Figure : Toy noise problem

Laurent Duval: Sparsity 101: Statistical estimators Central location and dispersion IFP Energies nouvelles

Introduction Estimators Derivations Wavelet use

2/7

IntroductionWhat is the trend? Where is the outlier?

0 2 4 6 8 10

−2

0

2

4

6

8

10

12

14

16

18

20

0

Figure : Toy noise problem

Laurent Duval: Sparsity 101: Statistical estimators Central location and dispersion IFP Energies nouvelles

Introduction Estimators Derivations Wavelet use

2/7

IntroductionWhat is the trend? Where is the outlier?

0 2 4 6 8 10

−2

0

2

4

6

8

10

12

14

16

18

20

0.5

Figure : Toy noise problem

Laurent Duval: Sparsity 101: Statistical estimators Central location and dispersion IFP Energies nouvelles

Introduction Estimators Derivations Wavelet use

2/7

IntroductionWhat is the trend? Where is the outlier?

0 2 4 6 8 10

−2

0

2

4

6

8

10

12

14

16

18

20

−1

Figure : Toy noise problem

Laurent Duval: Sparsity 101: Statistical estimators Central location and dispersion IFP Energies nouvelles

Introduction Estimators Derivations Wavelet use

3/7

Standard estimators

Compute a representative central location m ?

◮ mean: m = 1N

∑

xi

Laurent Duval: Sparsity 101: Statistical estimators Central location and dispersion IFP Energies nouvelles

Introduction Estimators Derivations Wavelet use

3/7

Standard estimators

Compute a representative central location m ?

◮ mean: m = 1N

∑

xi

◮ median: #(xi < m) = #(xi > m) (N+12 position)

Laurent Duval: Sparsity 101: Statistical estimators Central location and dispersion IFP Energies nouvelles

Introduction Estimators Derivations Wavelet use

3/7

Standard estimators

Compute a representative central location m ?

◮ mean: m = 1N

∑

xi

◮ median: #(xi < m) = #(xi > m)

◮ mid-range: 12(min xi +max xi )

Laurent Duval: Sparsity 101: Statistical estimators Central location and dispersion IFP Energies nouvelles

Introduction Estimators Derivations Wavelet use

3/7

Standard estimators

Compute a representative central location m ?

◮ mean: m = 1N

∑

xi

◮ median: #(xi < m) = #(xi > m)

◮ mid-range: 12(min xi +max xi )

◮ mid-hinge: 12(Q1(xi ) + Q3(xi ))

Laurent Duval: Sparsity 101: Statistical estimators Central location and dispersion IFP Energies nouvelles

Introduction Estimators Derivations Wavelet use

3/7

Standard estimators

Compute a representative central location m ?

◮ mean: m = 1N

∑

xi

◮ median: #(xi < m) = #(xi > m)

◮ mid-range: 12(min xi +max xi )

◮ mid-hinge: 12(Q1(xi ) + Q3(xi ))

◮ mode: argmax p(x)

Laurent Duval: Sparsity 101: Statistical estimators Central location and dispersion IFP Energies nouvelles

Introduction Estimators Derivations Wavelet use

3/7

Standard estimators

Compute a representative central location m ?

◮ mean: m = 1N

∑

xi

◮ median: #(xi < m) = #(xi > m)

◮ mid-range: 12(min xi +max xi )

◮ mid-hinge: 12(Q1(xi ) + Q3(xi ))

◮ mode: argmax p(x)

◮ even: min xi , max xi (and anything else in between?)

Laurent Duval: Sparsity 101: Statistical estimators Central location and dispersion IFP Energies nouvelles

Introduction Estimators Derivations Wavelet use

3/7

Standard estimators

Compute a representative central location m ?

◮ mean: m = 1N

∑

xi

◮ median: #(xi < m) = #(xi > m)

◮ mid-range: 12(min xi +max xi )

◮ mid-hinge: 12(Q1(xi ) + Q3(xi ))

◮ mode: argmax p(x)

◮ even: min xi , max xi◮ arbitrary algorithmic choice?

Laurent Duval: Sparsity 101: Statistical estimators Central location and dispersion IFP Energies nouvelles

Introduction Estimators Derivations Wavelet use

3/7

Standard estimators

Compute a representative central location m ?

◮ mean: m = 1N

∑

xi

◮ median: #(xi < m) = #(xi > m)

◮ mid-range: 12(min xi +max xi )

◮ mid-hinge: 12(Q1(xi ) + Q3(xi ))

◮ mode: argmax p(x)

◮ even: min xi , max xi◮ arbitrary algorithmic choice?

◮ no, answer to an optimization problem

Laurent Duval: Sparsity 101: Statistical estimators Central location and dispersion IFP Energies nouvelles

Introduction Estimators Derivations Wavelet use

3/7

Standard estimators

Compute a representative central location m ?

◮ mean: m = 1N

∑

xi ← argmin∑

(xi −m)2

◮ median: #(xi < m) = #(xi > m)

◮ mid-range: 12(min xi +max xi )

◮ mid-hinge: 12(Q1(xi ) + Q3(xi ))

◮ mode: argmax p(x)

◮ even: min xi , max xi◮ arbitrary algorithmic choice?

◮ no, answer to an optimization problem

Laurent Duval: Sparsity 101: Statistical estimators Central location and dispersion IFP Energies nouvelles

Introduction Estimators Derivations Wavelet use

3/7

Standard estimators

Compute a representative central location m ?

◮ mean: m = 1N

∑

xi ← argmin∑

wi (xi −m)2

◮ median: #(xi < m) = #(xi > m)

◮ mid-range: 12(min xi +max xi )

◮ mid-hinge: 12(Q1(xi ) + Q3(xi ))

◮ mode: argmax p(x)

◮ even: min xi , max xi◮ arbitrary algorithmic choice?

◮ no, answer to an optimization problem

Laurent Duval: Sparsity 101: Statistical estimators Central location and dispersion IFP Energies nouvelles

Introduction Estimators Derivations Wavelet use

3/7

Standard estimators

Compute a representative central location m ?

◮ mean: m = 1N

∑

xi ← argmin∑

(xi −m)2

◮ median: #(xi < m) = #(xi > m) ← argmin∑

|xi −m|

◮ mid-range: 12(min xi +max xi )

◮ mid-hinge: 12(Q1(xi ) + Q3(xi ))

◮ mode: argmax p(x)

◮ even: min xi , max xi◮ arbitrary algorithmic choice?

◮ no, answer to an optimization problem

Laurent Duval: Sparsity 101: Statistical estimators Central location and dispersion IFP Energies nouvelles

Introduction Estimators Derivations Wavelet use

3/7

Standard estimators

Compute a representative central location m ?

◮ mean: m = 1N

∑

xi ← argmin∑

(xi −m)2

◮ median: #(xi < m) = #(xi > m) ← argmin∑

wi |xi −m|

◮ mid-range: 12(min xi +max xi )

◮ mid-hinge: 12(Q1(xi ) + Q3(xi ))

◮ mode: argmax p(x)

◮ even: min xi , max xi◮ arbitrary algorithmic choice?

◮ no, answer to an optimization problem

Laurent Duval: Sparsity 101: Statistical estimators Central location and dispersion IFP Energies nouvelles

Introduction Estimators Derivations Wavelet use

3/7

Standard estimators

Compute a representative central location m ?

◮ mean: m = 1N

∑

xi ← argmin∑

(xi −m)2

◮ median: #(xi < m) = #(xi > m) ← argmin∑

|xi −m|

◮ mid-range: 12(min xi +max xi ) ← argminmax |xi −m|

◮ mid-hinge: 12(Q1(xi ) + Q3(xi ))

◮ mode: argmax p(x)

◮ even: min xi , max xi◮ arbitrary algorithmic choice?

◮ no, answer to an optimization problem

Laurent Duval: Sparsity 101: Statistical estimators Central location and dispersion IFP Energies nouvelles

Introduction Estimators Derivations Wavelet use

3/7

Standard estimators

Compute a representative central location m ?

◮ mean: m = 1N

∑

xi ← argmin∑

(xi −m)2

◮ median: #(xi < m) = #(xi > m) ← argmin∑

|xi −m|

◮ mid-range: 12(min xi +max xi ) ← argminmax |xi −m|

◮ mid-hinge: 12(Q1(xi ) + Q3(xi )) ← argminmed|xi −m|

◮ mode: argmax p(x)

◮ even: min xi , max xi◮ arbitrary algorithmic choice?

◮ no, answer to an optimization problem

Laurent Duval: Sparsity 101: Statistical estimators Central location and dispersion IFP Energies nouvelles

Introduction Estimators Derivations Wavelet use

3/7

Standard estimators

Compute a representative central location m ?

◮ mean: m = 1N

∑

xi

◮ median: #(xi < m) = #(xi > m)

◮ mid-range

◮ mid-hinge

◮ mode: argmax p(x)

◮ even: min xi , max xi◮ arbitrary algorithmic choice?

◮ no, answer to an optimization problem

◮ with a natural spread

Laurent Duval: Sparsity 101: Statistical estimators Central location and dispersion IFP Energies nouvelles

Introduction Estimators Derivations Wavelet use

3/7

Standard estimators

Compute a representative dispersion/spread ?

◮ mean: m = 1N

∑

xi

◮ median: #(xi < m) = #(xi > m)

◮ mid-range

◮ mid-hinge

◮ mode: argmax p(x)

◮ even: min xi , max xi◮ arbitrary algorithmic choice?

◮ no, answer to an optimization problem

◮ with a natural spread

Laurent Duval: Sparsity 101: Statistical estimators Central location and dispersion IFP Energies nouvelles

Introduction Estimators Derivations Wavelet use

3/7

Standard estimators

Compute a representative dispersion/spread ?

◮ mean: m = 1N

∑

xi → σ =√

1N

∑

(xi −m)2

◮ median: #(xi < m) = #(xi > m)

◮ mid-range

◮ mid-hinge

◮ mode: argmax p(x)

◮ even: min xi , max xi◮ arbitrary algorithmic choice?

◮ no, answer to an optimization problem

◮ with a natural spread

Laurent Duval: Sparsity 101: Statistical estimators Central location and dispersion IFP Energies nouvelles

Introduction Estimators Derivations Wavelet use

3/7

Standard estimators

Compute a representative dispersion/spread ?

◮ mean: m = 1N

∑

xi → σ =√

1N

∑

(xi −m)2

◮ median: #(xi < m) = #(xi > m) → MAD = med|xi −m|

◮ mid-range

◮ mid-hinge

◮ mode: argmax p(x)

◮ even: min xi , max xi◮ arbitrary algorithmic choice?

◮ no, answer to an optimization problem

◮ with a natural spread

Laurent Duval: Sparsity 101: Statistical estimators Central location and dispersion IFP Energies nouvelles

Introduction Estimators Derivations Wavelet use

3/7

Standard estimators

Compute a representative dispersion/spread ?

◮ mean: m = 1N

∑

xi → σ =√

1N

∑

(xi −m)2

◮ median: #(xi < m) = #(xi > m) → MAD = med|xi −m|

◮ mid-range → 12(max xi −min xi ) (half-range, mid-span)

◮ mid-hinge

◮ mode: argmax p(x)

◮ even: min xi , max xi◮ arbitrary algorithmic choice?

◮ no, answer to an optimization problem

◮ with a natural spread

Laurent Duval: Sparsity 101: Statistical estimators Central location and dispersion IFP Energies nouvelles

Introduction Estimators Derivations Wavelet use

3/7

Standard estimators

Compute a representative dispersion/spread ?

◮ mean: m = 1N

∑

xi → σ =√

1N

∑

(xi −m)2

◮ median: #(xi < m) = #(xi > m) → MAD = med|xi −m|

◮ mid-range → 12(max xi −min xi ) (half-range, mid-span)

◮ mid-hinge → 12(Q3(xi )− Q1(xi )) (IQR/mid-spread)

◮ mode: argmax p(x)

◮ even: min xi , max xi◮ arbitrary algorithmic choice?

◮ no, answer to an optimization problem

◮ with a natural spread

Laurent Duval: Sparsity 101: Statistical estimators Central location and dispersion IFP Energies nouvelles

Introduction Estimators Derivations Wavelet use

3/7

Standard estimators

Compute a representative dispersion/spread ?

◮ mean: m = 1N

∑

xi → σ =√

1N

∑

(xi −m)2

◮ median: #(xi < m) = #(xi > m) → MAD = med|xi −m|

◮ mid-range → 12(max xi −min xi ) (half-range, mid-span)

◮ mid-hinge → 12(Q3(xi )− Q1(xi )) (IQR/mid-spread)

◮ mode: argmax p(x)

◮ even: min xi , max xi◮ arbitrary algorithmic choice?

◮ no, answer to an optimization problem

◮ with a natural spread

Bonuses: add shape factor (skewness, kurtosis); Q1+2×med+Q34 . . .

Laurent Duval: Sparsity 101: Statistical estimators Central location and dispersion IFP Energies nouvelles

Introduction Estimators Derivations Wavelet use

4/7

Standard estimators: ℓ2

Least-squares estimator of weighted central location:

f (m) =∑

wi (xi −m)2

df

dm=

∑

−2wi (xi −m)

df

dm= 0⇔

∑

−2wixi =∑

−2wim

∑

wixi =∑

wim

m =

∑

wixi∑

wi

Weighted sum location → all linear filters; Gaussian noise

Laurent Duval: Sparsity 101: Statistical estimators Central location and dispersion IFP Energies nouvelles

Introduction Estimators Derivations Wavelet use

5/7

Standard estimators: ℓ1

Least-magnitude estimator of (weighted) central location:

f (m) =∑

wi |xi −m|

df

dm≈

∑

−wi sign(xi −m)

df

dm= 0⇔ #wi

(xi < m) = #wi(xi > m)

Equality reached when m stands “in between”

m = medianwixi

Sorted location → gen. weighted median filters; Laplace noise

Laurent Duval: Sparsity 101: Statistical estimators Central location and dispersion IFP Energies nouvelles

Introduction Estimators Derivations Wavelet use

6/7

Other (less standard) estimatorsImportance of measurement units

◮ harmonic, geometric, arithmetico-geometric means100 km at 150 kmh−1, 100 km at 100 kmh−1 → 120 kmh−1

◮ M-estimators, L-estimators, robust statistics

◮ no natural dispersion in general

◮ time vs individuals; 1D/2D; representative scale; transforms

A robust Gaussian noise dispersion estimator (details)

σ ≃median|ci |

0.6745

Use with wavelet shrinkage operators (soft , hard, garrote, etc.)

ci = S(ci ,Λ(σ))

Laurent Duval: Sparsity 101: Statistical estimators Central location and dispersion IFP Energies nouvelles

Introduction Estimators Derivations Wavelet use

7/7

Noise level estimation and wavelets

Figure : Noise estimation with standard 1D wavelets

Laurent Duval: Sparsity 101: Statistical estimators Central location and dispersion IFP Energies nouvelles

Introduction Estimators Derivations Wavelet use

7/7

Noise level estimation and wavelets

50 100 150 200 250

−0.5

0

0.5

1

1.5

2

Am

plitu

de

Clean trace/Noisy trace (std:0.00125)50 100 150 200 250

−0.5

0

0.5

1

1.5

2

Am

plitu

de

Clean trace/Noisy trace (std:0.0025)50 100 150 200 250

−0.5

0

0.5

1

1.5

2

Am

plitu

de

Clean trace/Noisy trace (std:0.00625)50 100 150 200 250

−0.5

0

0.5

1

1.5

2

Am

plitu

de

Clean trace/Noisy trace (std:0.00875)50 100 150 200 250

−0.5

0

0.5

1

1.5

2

Am

plitu

de

Clean trace/Noisy trace (std:0.0125)

50 100 150 200 250

−0.5

0

0.5

1

1.5

2

Am

plitu

de

Clean trace/Noisy trace (std:0.025)50 100 150 200 250

−0.5

0

0.5

1

1.5

2

Am

plitu

de

Clean trace/Noisy trace (std:0.0625)50 100 150 200 250

−0.5

0

0.5

1

1.5

2

Am

plitu

de

Clean trace/Noisy trace (std:0.0875)50 100 150 200 250

−0.5

0

0.5

1

1.5

2

Am

plitu

de

Clean trace/Noisy trace (std:0.125)50 100 150 200 250

−0.5

0

0.5

1

1.5

2

Am

plitu

de

Clean trace/Noisy trace (std:0.15)

50 100 150 200 250−1

0

1

2

Am

plitu

de

Clean trace/Noisy trace (std:0.1875)50 100 150 200 250

−1

0

1

2

Am

plitu

de

Clean trace/Noisy trace (std:0.2125)50 100 150 200 250

−1

0

1

2

Am

plitu

de

Clean trace/Noisy trace (std:0.25)50 100 150 200 250

−1

0

1

2

Am

plitu

deClean trace/Noisy trace (std:0.3125)

50 100 150 200 250

−1

0

1

2

Am

plitu

de

Clean trace/Noisy trace (std:0.5)

Figure : Noise estimation with standard 1D wavelets

Laurent Duval: Sparsity 101: Statistical estimators Central location and dispersion IFP Energies nouvelles

Introduction Estimators Derivations Wavelet use

7/7

Noise level estimation and wavelets

20 40 60 80 100 120

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

Am

plitu

de

Noisy coefficients (std:0.00125)20 40 60 80 100 120

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

Am

plitu

de

Noisy coefficients (std:0.0025)20 40 60 80 100 120

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

Am

plitu

de

Noisy coefficients (std:0.00625)20 40 60 80 100 120

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

Am

plitu

de

Noisy coefficients (std:0.00875)20 40 60 80 100 120

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

Am

plitu

de

Noisy coefficients (std:0.0125)

20 40 60 80 100 120

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

Am

plitu

de

Noisy coefficients (std:0.025)20 40 60 80 100 120

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

Am

plitu

de

Noisy coefficients (std:0.0625)20 40 60 80 100 120

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

Am

plitu

de

Noisy coefficients (std:0.0875)20 40 60 80 100 120

−0.5

0

0.5

Am

plitu

de

Noisy coefficients (std:0.125)20 40 60 80 100 120

−0.4

−0.2

0

0.2

0.4

0.6

0.8

Am

plitu

de

Noisy coefficients (std:0.15)

20 40 60 80 100 120

−0.5

0

0.5

1

Am

plitu

de

Noisy coefficients (std:0.1875)20 40 60 80 100 120

−0.4

−0.2

0

0.2

0.4

0.6

0.8

Am

plitu

de

Noisy coefficients (std:0.2125)20 40 60 80 100 120

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

Am

plitu

de

Noisy coefficients (std:0.25)20 40 60 80 100 120

−1

−0.5

0

0.5

Am

plitu

deNoisy coefficients (std:0.3125)

20 40 60 80 100 120

−1

−0.5

0

0.5

1

1.5

Am

plitu

de

Noisy coefficients (std:0.5)

Figure : Noise estimation with standard 1D wavelets

Laurent Duval: Sparsity 101: Statistical estimators Central location and dispersion IFP Energies nouvelles

Introduction Estimators Derivations Wavelet use

7/7

Noise level estimation and wavelets

−0.1 0 0.1 0.2 0.3 0.4 0.5 0.60

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Input noise std

Wav

elet

dom

ain

nois

e es

timat

ion

Figure : Noise estimation with standard 1D wavelets

Laurent Duval: Sparsity 101: Statistical estimators Central location and dispersion IFP Energies nouvelles