BMI2 SS07 – Class 3 “Image Processing 1” Slide 1 Biomedical Imaging 2 Class 3,4 –...

BMI2 SS07 – Class 3 “Image Processing 1” Slide 1

Biomedical Imaging 2Biomedical Imaging 2

Class 3,4 – Regularization; Time Series Analysis (Pt. 1); Image Post-processing (Pt. 1)

02/06/07, 02/13/07


Well-Posedness, Ill-PosednessWell-Posedness, Ill-Posedness

• Definition due to Hadamard, 1915: Given the mapping A: X→Y, the equation Ax = y is well-posed if– (Existence) For every y in Y, there is an x in X such

that Ax = y.

– (Uniqueness) If Ax1 = Ax2, then x1 = x2.

– (Stability) A-1 is continuous• x = A-1y

• A-1(y + dy) = x + dx

• Ax = y is ill-posed if it is not well-posed


Well- and Ill-conditioned ProblemsWell- and Ill-conditioned Problems

• Overdetermined linear systems (more equations than unknowns) are ill-posed, strictly speaking– No exact Solution exists!– Existence is imposed by using least-squares solution

• Underdetermined linear systems (fewer equations than unknowns) are ill-posed, strictly speaking– Infinitely many Solutions exist!– Uniqueness is imposed by using minimum-norm

solution

• Can a discrete linear system be unstable?


Well- and Ill-conditioned ProblemsWell- and Ill-conditioned Problems

• Can a discrete linear system be unstable?– Strictly speaking, no!

• A-1(x0 + x) = A-1x0 + A-1∙x

• All elements of A-1 and of x are finite• Therefore, all elements of A-1∙x must be finite

• However, it certainly can be true that||A-1∙x||/||A-1x0|| >> ||x||/||x0||

– Small change in input → large change in output– Such a system is called ill-conditioned


Example of Ill-conditioningExample of Ill-conditioning

1 1 12 3 4

1 1 1 12 3 4 5

1 1 1 13 4 5 6

1 1 1 14 5 6 7

14×4 Hilbert matrix:

This is a full-rank, non-singular matrix, and so it has a well-defined inverse:

16 120 240 140

120 1200 2700 1680

240 2700 6480 4200

140 1680 4200 2800



1 1 12 3 4

1 1 1 12 3 4 51 1 1 13 4 5 61 1 1 14 5 6 7

1 12

1 16 120 240 140

120 1200 2700 1680

240 2700 6480 4200

140 1680 4200 2800

116 120 240 140

120 1200 2700 1680

240 2700 6480 4200

140 1680 4200 2800

13 4

1 1 1 12 3 4 51 1 1 13 4 5 61 1 1 14 5 6 7

1 0 0 0

0 1 0 0

0 0 1 0

0 0 0 1

Positive and negative products cancel in exactly the right manner:

But what happens if we change any element by even a small amount?



1 1 12 3 4

1 1 1 112 3 4 5

1 1 1 13 4 5 61 1 1 14 5 6 7

1 16 120 240 140 1

120 1200 2700 1680 1, ,

240 2700 6480 4200 1

140 1680 4200 2800 1

A A y

1

16 120 240 140 1 4

120 1200 2700 1680 1 60

240 2700 6480 4200 1 180

140 1680 4200 2800 1 140

x A y

1 0.0043256 0.99567

1 0.016656 0.98334But what if ?

1 0.0012533 1.0013

1 0.0028768 1.0029

y



1

1

16 120 240 140 1 4

120 1200 2700 1680 1 60

240 2700 6480 4200 1 180

140 1680 4200 2800 1 140

16 120 240 140

120 1200 2700 1680

240 2700 6480 4200

140 1680 4200 2800

y

x A y

x x A y

0.99567

0.98334

1.0013

1.0029

2.1725

41.981

140.03

115.41



1 1 1

2 2 2

3 3 3

4 4 4

0,0.011 4

0,0.011 60, ~

0,0.011 180

0,0.011 140

Nx y x

Nx y x

Nx y x

Nx y x

1 1

2 2

3 3

4 4

Sample 10,000 times:

0.000142 0.010053

0.000005 0.009951,

0.000207 0.010162

0.000006 0.009986

x x

x xmean std

x x

x x

1 1

2 2

3 3

4 4

0.05 3.05

0.57 34.2,

1.36 82.3

0.88 53.5

y y

y ymean std

y y

y y



1 1 1

2 2 2

3 3 3

4 4 4

1

2

3

4

0,0.011 4

0,0.011 60, ~ '

0,0.011 180

0,0.011 140

0.05

0.57

1.36

0.

Nx y x

Nx y x

Nx y x

Nx y x

y

ymean

y

y

1

2

3

4

3.05

34.2,

82.3

88 53.5

y

ystd

y

y

Our “image reconstruction” operator is unbiased

But it has high variance


Tradeoff Between Bias and VarianceTradeoff Between Bias and Variance

The numerical values in y are eventually represented as gray levels or colors in an image:

4

60

180

140

-4

-180

60

140

As long as the color pattern makes an interpretable image, do you care if the numerical values are exactly right?

That is, are you willing to give up accuracy to gain precision (i.e., decrease variance by increasing bias)?


RegularizationRegularization

• For overdetermined systems, we define the pseudo-inverse A+ = (ATA)-1AT.

• For underdetermined systems, we define the pseudo-inverse A+ = AT(AAT)-1.

• For the Hilbert matrix, either of the preceding reduces to the true inverse:– (ATA)-1AT = [A-1(AT)-1]AT = A-1[(AT)-1AT] = A-1

– AT(AAT)-1 = AT[(AT)-1A-1] = [AT(AT)-1 ]A-1 = A-1

• Now we introduce one additional term:– A+ = (ATA + αI)-1AT

– A+ = AT(AAT + αI)-1Regularization term

Regularization parameter


What Does Regularization Do?What Does Regularization Do?

• Now we introduce one additional term:– A+ = (ATA + αI)-1AT, A+ = AT(AAT + αI)-1

• This particular variety is called Tikhonov regularization– Imposes continuity on the computed y

• That is, limits the spatial scale on which solution can change (long-pass filter)

– Could replace the I in the regularization term with a discrete 1st, 2nd, etc., derivative operator

• Then continuity would be imposed on the corresponding derivative of the solution


Something To Watch Out forSomething To Watch Out for

• Two things that can go wrong when Tik. Reg. is used:– α is too small (under-regularized case): noise

continues to wreak havoc– α is too large (over-regularized case): ability to

capture spatial variations of interest is lost

• Is there an algorithm guaranteed to produce the optimal α?– Alas, no (when is life ever that easy?)– Special cases; Monte Carlo simulations; trial-and-

error


Regularized Hilbert Matrix InverseRegularized Hilbert Matrix Inverse

11 T T

16 120 240 140

120 1200 2700 1680

240 2700 6480 4200

140 1680 4200 2800

A A A A

1T T A A A I A

7

7.9301 29.208 21.508 2.0164

29.208 178.01 239.85 80.65310 :

21.508 239.85 556.97 349.04

2.0164 80.653 349.04 296.07

8

11.445 68.715 116.54 59.731

68.715 622.57 1309.8 776.1610 :

116.54 1309.8 3133 2023.9

59.731 776.16 2023.9 1385.1

9

15.148 110.41 216.92 124.99

110.41 1092.1 2440.1 151110 :

216.92 2440.1 5854.3 3793.2

124.99 1511 3793.2 2535.5


Impact of Noisy Data Impact of Noisy Data

1 1

2 2

3 3

4 4

0.05 3.05

0.57 34.2,

1.36 82.3

0.88 53.5

y y

y ymean std

y y

y y

Unregularized solution (i.e., α = 0):

1 1

2 29

3 3

4 4

0.71 2.76

7.96 30.910 : ,

19.1 74.4

12.4 48.4

y y

y ymean std

y y

y y

Conclusion: Under-regularized!



1 1

2 2

3 3

4 4

0.05 3.05

0.57 34.2,

1.36 82.3

0.88 53.5

y y

y ymean std

y y

y y


1 1

2 27

3 3

4 4

6.25 0.38

70.5 3.1410 : ,

170 7.05

110 4.66

y y

y ymean std

y y

y y

Conclusion: Over-regularized!



1 1

2 2

3 3

4 4

0.05 3.05

0.57 34.2,

1.36 82.3

0.88 53.5

y y

y ymean std

y y

y y


1 1

2 28

3 3

4 4

3.56 1.50

40.1 16.610 : ,

96.5 39.8

62.7 25.9

y y

y ymean std

y y

y y

Conclusion: Getting close?


Final Choice of α Parameter

7.8

1 1

2 2

3 3

4 4

-4

6010 : noise-free = ,

-180

140

0.32 1.16

11.3 12.8, .

62.9 30.6

63.8 19.9

y y

y ymean std

y y

y y

y


Other Types of RegularizationOther Types of Regularization

• Truncated singular value decomposition• Discrete-cosine transform• Statistical (Bayesian)

– Requires knowledge of the solution and noise covariances

• Iterative– Steepest descent– Conjugate-gradient descent– Richardson-Lucy– Landweber


Time Series Analysis…Time Series Analysis…

Definitions• The branch of quantitative forecasting in which data for one variable are

examined for patterns of trend, seasonality, and cycle. nces.ed.gov/programs/projections/appendix_D.asp

• Analysis of any variable classified by time, in which the values of the variable are functions of the time periods. www.indiainfoline.com/bisc/matt.html

• An analysis conducted on people observed over multiple time periods. www.rwjf.org/reports/npreports/hcrig.html

• A type of forecast in which data relating to past demand are used to predict future demand. highered.mcgraw-hill.com/sites/0072506369/student_view0/chapter12/glossary.html

• In statistics and signal processing, a time series is a sequence of data points, measured typically at successive times, spaced apart at uniform time intervals. Time series analysis comprises methods that attempt to understand such time series, often either to understand the underlying theory of the data points (where did they come from? what generated them?), or to make forecasts (predictions). en.wikipedia.org/wiki/Time_series_analysis



Varieties• Frequency (spectral) analysis

– Fourier transform: amplitude and phase– Power spectrum; power spectral density

• Auto-spectral density– Cross-spectral density– Coherence

• Correlation Analysis– Cross-correlation function

• Cross-covariance• Correlation coefficient function

– Autocorrelation function– Cross-spectral density

• Auto-spectral density



Varieties• Time-frequency analysis

– Short-time Fourier transform– Wavelet analysis

• Descriptive Statistics– Mean / median; standard deviation / variance / range– Short-time mean, standard deviation, etc.

• Forecasting / Prediction– Autoregressive (AR)– Moving Average (MA)– Autoregressive moving average (ARMA)– Autoregressive integrated moving average (ARIMA)

• Random walk, random trend• Exponential weighted moving average



Varieties• Signal separation

– Data-driven [blind source separation (BSS), signal source separation (SSS)]

• Principal component analysis (PCA)• Independent component analysis (ICA)• Extended spatial decomposition, extended temporal

decomposition• Canonical correlation analysis (CCA)• Singular-value decomposition (SVD) an essential

ingredient of all– Model-based

• General linear model (GLM)• Analysis of variance (ANOVA, ANCOVA, MANOVA, MANCOVA)

– e.g., Statistical Parametric Mapping, BrainVoyager, AFNI


A “Family Secret” of Time Series Analysis…A “Family Secret” of Time Series Analysis…• Scary-looking formulas, such as

– Are useful and important to learn at some stage, but not really essential for understanding how all these methods work

• All the math you really need to know, for understanding, is– How to add: 3 + 5 = 8, 2 - 7 = 2 + (-7) = -5– How to multiply: 3 × 5 = 15, 2 × (-7) = -14

• Multiplication distributes over addition

u × (v1 + v2 + v3 + …) = u×v1 + u×v2 + u×v3 + …

– Pythagorean theorem: a2 + b2 = c2

1 2 1 2

1, ,

2

, , ,

, , , ,

x y

i t i t

i x y

x y

x y x y

F f t e dt f t F e d

F f x y e dxdy

F f x y f x y F F

a

b

c


A “Family Secret” of Time Series Analysis…A “Family Secret” of Time Series Analysis…

A most fundamental mathematical operation for time series analysis:

1 2 3

1 2 3

, , , ...,

, , , ...,

N

N

x x x x

y y y y

31 2

31 2

1 2 3

, , , ...,

N

N

N

xx x x

yy y y

z z zz

1 2 3 ... Nz z z z Z

The xi time series is measurement or image data. The yi time series depends on what type of analysis we’re doing:

Fourier analysis: yi is a sinusoidal function

Correlation analysis: yi is a second data or image time series

Wavelet or short-time FT: non-zero yi values are concentrated in a small range of i, while most of the yis are 0.

GLM: yi is an ideal, or model, time series that we expect some of the xi time series to resemble


Example: Fourier AnalysisExample: Fourier Analysis

time [s]

0.0 0.5 1.0 1.5 2.0

Fun

ctio

n V

alue

-0.8

-0.6

-0.4

-0.2

0.0

0.2

0.4

0.6

0.8Individual Frequency Components



time [s]

0.0 0.5 1.0 1.5 2.0

Fun

ctio

n V

alue

-1.2

-0.8

-0.4

0.0

0.4

0.8

1.2Partial Sum 1

time [s]

0.0 0.5 1.0 1.5 2.0

Fun

ctio

n V

alue

-1.2

-0.8

-0.4

0.0

0.4

0.8

1.2Partial Sum 2

time [s]

0.0 0.5 1.0 1.5 2.0

Fun

ctio

n V

alue

-1.2

-0.8

-0.4

0.0

0.4

0.8

1.2Partial Sum 3

time [s]

0.0 0.5 1.0 1.5 2.0

Fun

ctio

n V

alue

-1.2

-0.8

-0.4

0.0

0.4

0.8

1.2Partial Sum 4



time [s]

0.0 0.5 1.0 1.5 2.0

Fun

ctio

n V

alue

-1.2

-0.8

-0.4

0.0

0.4

0.8

1.2Partial Sum 5

time [s]

0.0 0.5 1.0 1.5 2.0

Fun

ctio

n V

alue

-1.2

-0.8

-0.4

0.0

0.4

0.8

1.2Partial Sum 6

time [s]

0.0 0.5 1.0 1.5 2.0

Fun

ctio

n V

alue

-1.2

-0.8

-0.4

0.0

0.4

0.8

1.2Partial Sum 7

time [s]

0.0 0.5 1.0 1.5 2.0

Fun

ctio

n V

alue

-1.2

-0.8

-0.4

0.0

0.4

0.8

1.2Partial Sum 8



time [s]

0.0 0.5 1.0 1.5 2.0

Fun

ctio

n V

alue

-1.2

-0.8

-0.4

0.0

0.4

0.8

1.2Partial Sum 9

time [s]

0.0 0.5 1.0 1.5 2.0F

unct

ion

Val

ue-1.2

-0.8

-0.4

0.0

0.4

0.8

1.2Partial sum 10



time [s]

0.0 0.5 1.0 1.5 2.0

Fun

ctio

n V

alue

-0.8

-0.6

-0.4

-0.2

0.0

0.2

0.4

0.6


time [s]

0.0 0.5 1.0 1.5 2.0

Fun

ctio

n V

alue

-1.2

-0.8

-0.4

0.0

0.4

0.8

1.2Sawtooth Wave

= Σ

×

sin10πt

time [s]

0.0 0.5 1.0 1.5 2.0

Fun

ctio

n V

alue

-1.2

-0.8

-0.4

0.0

0.4

0.8

1.2Sawtooth*sin(10*pi*t)



time [s]

0.0 0.5 1.0 1.5 2.0

Fun

ctio

n V

alue

-0.6

-0.4

-0.2

0.0

0.2

0.4

0.6

0.8First Frequency Component * sin(10*pi*t)

time [s]

0.0 0.5 1.0 1.5 2.0

Fun

ctio

n V

alue

-0.4

-0.3

-0.2

-0.1

0.0

0.1

0.2

0.3

0.4Second Frequency Component * sin(10*pi*t)

time [s]

0.0 0.5 1.0 1.5 2.0

Func

tion

Val

ue

-0.25

-0.20

-0.15

-0.10

-0.05

0.00

0.05

0.10

0.15

0.20Third Frequency Component * sin(10*pi*t)

time [s]

0.0 0.5 1.0 1.5 2.0

Func

tion

Val

ue

-0.20

-0.15

-0.10

-0.05

0.00

0.05

0.10

0.15

0.20Fourth Frequency Component * sin(10*pi*t)



time [s]

0.0 0.5 1.0 1.5 2.0

Func

tion

Val

ue

-0.14

-0.10

-0.06

-0.02

0.02

0.06

0.10

0.14Fifth Frequency Component * sin(10*pi*t)

time [s]

0.0 0.5 1.0 1.5 2.0

Func

tion

Val

ue

-0.15

-0.10

-0.05

0.00

0.05

0.10

0.15Sixth Frequency Component * sin(10*pi*t)

time [s]

0.0 0.5 1.0 1.5 2.0

Func

tion

Val

ue

-0.10

-0.05

0.00

0.05

0.10Seventh Frequency Component * sin(10*pi*t)

time [s]

0.0 0.5 1.0 1.5 2.0

Func

tion

Val

ue

-0.08

-0.04

0.00

0.04

0.08Eighth Frequency Component * sin(10*pi*t)



time [s]

0.0 0.5 1.0 1.5 2.0

Func

tion

Val

ue

-0.08

-0.06

-0.04

-0.02

0.00

0.02

0.04

0.06

0.08Ninth Frequency Component * sin(10*pi*t)

time [s]

0.0 0.5 1.0 1.5 2.0Fu

nctio

n V

alue

-0.06

-0.04

-0.02

0.00

0.02

0.04

0.06Tenth Frequency Component * sin(10*pi*t)



time [s]

0.0 0.5 1.0 1.5 2.0

Func

tion

Val

ue

-0.14

-0.10

-0.06

-0.02

0.02

0.06

0.10


time [s]

0.0 0.5 1.0 1.5 2.0

Fun

ctio

n V

alue

-1.2

-0.8

-0.4

0.0

0.4

0.8

1.2Sawtooth*sin(10*pi*t)

= ΣΣ



time [s]

0.0 0.5 1.0 1.5 2.0

Func

tion

Val

ue

-0.14

-0.10

-0.06

-0.02

0.02

0.06

0.10


time [s]

0.0 0.5 1.0 1.5 2.0F

unct

ion

Val

ue-0.8

-0.6

-0.4

-0.2

0.0

0.2

0.4

0.6



Another “Family Secret” of Time Series Analysis…Another “Family Secret” of Time Series Analysis…• The second operation that is fundamental to myriad forms of time-

series analysis is singular value decomposition (SVD)• Variations of SVD underlie:

– Principal component analysis (PCA)– Independent component analysis (ICA)– Canonical correlation analysis (CCA)– Extended spatial/temporal decorrelation


Significance of angle between x and b

Given an arbitrary N×N matrix A and N×1 vector x: ordinarily, b = Ax is different from x in both magnitude and direction.

x

b

However, for any A there will always be some particular directions such that b will be parallel to x (i.e., b is a simple scalar multiple of x, or Ax = λx) if x lies in one of these directions.An x that satisfies Ax = λx is an eigenvector, and λ is the corresponding eigenvalue.


Homogeneous Linear System: Ax = 0Homogeneous Linear System: Ax = 0

Recall definition of eigenvectors and eigenvalues:

Ax = λx, x 0.

Then Ax - λx = Ax - λIx = (A - λI)x = 0.

That is, the eigenvalues are those specific values of λ for which the matrix A - λI is singular, and the eigenvectors are the corresponding nullspaces.


Significance of angle between x and b

2 2

1 1 12

2 2

1 1 12

2 2

2 1

1 2

2 12 1 4 3

1 2

3,1

1 1 13: 0

1 1 1

1 1 11: 0

1 1

0

1

u u

u u

v v

v v

M

2 1 1 1

1 2

2 1 1 3

1 2 1

1 1

3

1 12 2

1 1 1 12 2 2 2

1 12 21 1

2 21 1

2 2

13

2

2 1 1 4

1 2 2 5

2 1 2 13 3 3

1 2 1 2

49 9

5

u v

u v u v

u v


Significance of eigenvalues and eigenvectors

An N×N A always has N eigenvalues.

If A is symmetric, and λ1 and λ2 are two distinct eigenvalues, the corresponding eigenvectors x1 and x2 are necessarily orthogonal.

If λ1 = λ2, we can always subtract off x1’s projection onto x2 from x1 (Gram-Schmidt orthogonalization).

If A is not symmetric, then its eigenvectors generally are not mutually orthogonal. But recall that the matrices AAT and ATA are always symmetric.

The square roots of the eigenvalues of AAT or ATA are thesingular values of A. The eigenvectors of AAT or ATA are thesingular vectors of A.

Computation of the eigenvalues and eigenvectors of AAT and ATA underlies a very useful linear algebraic technique called singular value decomposition (SVD).

SVD is the method that allows us to, among other things, tackle the one case we have not yet seen an explicit example of: finding the “solution” of a linear system when A is not of full rank.



1 1 1 12 2 2 2 T

1 1 1 12 2 2 2

2 1 3 0

1 2 0 1

USV

M is symmetric, so MTM = MMT

1

1 1T T 1 1

1 1 1 112 2 2 23

1 1 1 12 2 2 2

2 13 3

1 23 3

2 1

1 2

0

0 1

USV V S U

An orthogonal matrix is very easy to invert:

X-1 = XT

A diagonal matrix is very easy to invert:

just reciprocate each diagonal element



1 1 12 3 4

1 1 1 12 3 4 5 1

1 1 1 13 4 5 6

1 1 1 14 5 6 7

1 16 120 240 140

120 1200 2700 1680,

240 2700 6480 4200

140 1680 4200 2800

A A

Eigenvalues of A: 1.5002, 0.16914, 0.0067383, 9.6702×10-5

Eigenvalues of A-1: 0.66657, 5.9122, 148.41, 10341

Any arbitrary vector x is equal to a sum of the eigenvectors of A-1:

x = av1 + bv2 + cv3 + dv4, for some numbers a, b, c, d.

So A-1x = aA-1v1 + bA-1v2 + cA-1v3 + dA-1v4

= 0.66657av1 + 5.9122bv2 + 148.41cv3 + 10341dv4



1 1 12 3 4

1 1 1 1T2 3 4 5

4 4 4 41 1 1 13 4 5 6

51 1 1 14 5 6 7

1 1.5002 0 0 0

0 0.16914 0 0

0 0 0.0067383 0

0 0 0 9.6702 10

A U V

T4 3 3 4

1.5002 0 0

* 0 0.16914 0

0 0 0.0067383

1 0.5 0.33333 0.25

0.5 0.33332 0.25003 0.19998

0.33333 0.25003 0.19994 0.16671

0.25 0.19998 0.16671 0.14283

A U V



1 T4 3 3 4

0.66657 0 0

* 0 5.9122 0

0 0 148.41

7.1869 20.766 1.082 15.338

20.766 0.33332 9.8223 69.076

1.082 9.8223 3.0954 11.097

15.338 69.076 11.097 62.066

A V U

1 0.5 0.33333 0.25

0.5 0.33332 0.25003 0.19998*

0.33333 0.25003 0.19994 0.16671

0.25 0.19998 0.16671 0.14283

A

1

16 120 240 140

120 1200 2700 1680

240 2700 6480 4200

140 1680 4200 2800

A


Regularization Redux

1 1 12 3 4

1 1 1 12 3 4 5

1 1 1 13 4 5 6

1 1 1 14 5 6 7

1 1 2.0833 16 120 240 140 2.0833

1 1.2833 120 1200 2700 1680 1.2833,

1 0.95 240 2700 6480 4200 0.95

1 0.75952 140 1680 4200 2800 0.75952

1

1

1

1

16 120 240 140 2.0594 1.5351

120 1200 2700 1680 1.2986 8.776But...

240 2700 6480 4200 0.9613 28.408

140 1680 4200 2800 0.75924 18.228

7.1869 20.766 1.082 15.338 2.0833 3.2315

20.766 0.33332 9.8223 69.076 1.2833 1.8588

1.082 9.8223 3.0954 11.097 0.95 1.3319

15.338 69.076 11.097 62.066 0.75952 1.0443


Regularization Redux

16 120 240 140 2.0833 1

120 1200 2700 1680 1.2833 1

240 2700 6480 4200 0.95 1

140 1680 4200 2800 0.75952 1

16 120 240 140

120 1200 2700 1680

240 2700 6480 4200

140 1680 4200 2800

2.0594 1.5351

1.2986 8.776

0.9613 28.408

0.75924 18.228

7.1869 20.766 1.082 15.338 2.0833 3.2315

20.766 0.33332 9.8223 69.076 1.2833 1.8588

1.082 9.8223 3.0954 11.097 0.95 1.3319

15.338 69.076 11.097 62.066 0.75952 1.0443

7.1869 20.766 1.

082 15.338 2.0594 3.219

20.766 0.33332 9.8223 69.076 1.2986 1.8548

1.082 9.8223 3.0954 11.097 0.9613 1.3299

15.338 69.076 11.097 62.066 0.75924 1.0433


What happens if we try to use Gaussian elimination to solve Ax = b, but A is singular?

1 1 1 5

2 3 4 2

4 6 8 9

u

v

w

1 1 1 5

0 1 2 12

0 0 0 13

u

v

w

After second round of elimination:

There is no Solution!

These two equations are inconsistent.

Gaussian Elimination Redux

But there is a pseudoinverse, A+, which we can find by using SVD:

11 1 16 10 5

13

7 1 16 10 5

1 1 1

2 3 4 0 0

4 6 8


TT T

T

T T

,

1 1 1 0.13766 0.9904812.157 0 0.37568 0.55786 0.74004

2 3 4 0.44296 0.0615640 0.45053 0.83198 0.14875 0.53449

4 6 8 0.88591 0.12313

A AAA A A

AA

V xSA U x

How do we compute A+?

11 1 16 10 5

112.157 1

31

0.45053 7 1 16 10 5

T

0.37568 0.831980 0.13766 0.44296 0.88591

0.55786 0.14875 0 00 0.99048 0.061564 0.12313

0.74004 0.53449

S UV

A


As indicated, for this case AA+ ≠ I and AA+ ≠ I:


11 1 16 10 5

1 1 23 5 5

7 1 1 2 46 10 5 5 5

511 1 1 1 16 10 5 6 3 6

1 1 1 13 3 3 3

7 51 1 1 16 10 5 6 3 6

1 1 1 1 0 0

2 3 4 0 0 0

4 6 8 0

1 1 1

0 0 2 3 4

4 6 8

What is the pseudoinverse “solution,” and what is its significance?

227 22711 1 16 10 5 30 30

5 5 1613 3 3 5

7 127 127 321 16 10 5 30 30 5

5 1 1 1 5 5

0 0 2 2 3 4 2

9 4 6 8 9

x A b b Ax


We are not surprised that b+ ≠ b, because we already knew that the original system has no Solution.


That is, that no linear combination of the columns of A is equal to b.

165

325

2 2 216 32

5 5

1695

5 5

2

9

5 5 2 9

5.8138

Ax b b b

However, the “solution” x+ gives us that linear combination of columns of A which is closest to b, in the sense of minimizing the distance between Ax and b.


Example: Image Time-series Analysis via PCA

1 2

1 1 111 22 2 221 23 3 331 2

1 2

N

N

N

N

M M MMN

p p p

x x xt

x x xt

x x xt

x x xt

1 1 1 1 11 22 2 2 2 21 2

1 2 33 3 3 3 31 2

1 2 3

principal-componentimages

1 2

principal-componenttime series

1 0

0 2

N

NN

NN

M M M M MN

x x x

x x xq q q qs

x x xr r r rs

x x x


METHODS: Target MediumMETHODS: Target Medium

Quasiperiodic

Chaotic (Hénon attractor)

Stochastic

Chaotic (Hénon attractor)

Indicated dynamics were imposed on the inclusions’ μa, which ranged from 0.048 cm-1 to 0.072 cm-1 over time. The remainder of the target had a constant μa of 0.06 cm-1, and the entire target had constant μs = 10 cm-1. Black dots denote source/detector locations.

8 cm

0.6 cm


METHODS: Dynamics ModelsMETHODS: Dynamics Models

y1(t)

y3(t)

y2(t)

y4(t)

Quasiperiodic Chaos 1

Chaos 2 Uniform Stochastic


RESULTS: Statistics of Image Time SeriesRESULTS: Statistics of Image Time Series

-1.5

-1

-0.5

0

0.5

1

1.5x 10

-9Position-dependent Temporal Mean

10 20 30 40

10

20

30

40

0 200 400 600 800 1000-1

-0.5

0

0.5

1x 10

-4 Time-dependent Spatial Mean

0 10 20 3010

-8

10-6

10-4

10-2

100

Singular Values

0 10 20 3050

60

70

80

90

100Cumulative % of Variability


RESULTS: Image Time Series PCARESULTS: Image Time Series PCA

0 20 40 60 80 100 120 140 160 180 200-0.08

-0.06

-0.04

-0.02

0

0.02

0.04

0.06

0.08

0 20 40 60 80 100 120 140 160 180 200

-0.06

-0.04

-0.02

0

0.02

0.04

0.06

0 20 40 60 80 100 120 140 160 180 200

-0.06

-0.04

-0.02

0

0.02

0.04

0.06

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

5 10 15 20 25 30 35 40

5

10

15

20

25

30

35

40

-0.05

0

0.05

0.1

0.15

0.2

0.25

5 10 15 20 25 30 35 40

5

10

15

20

25

30

35

40

-0.05

0

0.05

0.1

0.15

0.2

0.25

5 10 15 20 25 30 35 40

5

10

15

20

25

30

35

40

-0.15

-0.1

-0.05

0

0.05

0.1

0.15

5 10 15 20 25 30 35 40

5

10

15

20

25

30

35

40

-0.15

-0.1

-0.05

0

0.05

0.1

0.15

0.2

5 10 15 20 25 30 35 40

5

10

15

20

25

30

35

40

0 20 40 60 80 100 120 140 160 180 200

-0.1

-0.08

-0.06

-0.04

-0.02

0

0.02

0.04

0.06

0.08

0 20 40 60 80 100 120 140 160 180 200

-0.08

-0.06

-0.04

-0.02

0

0.02

0.04

0.06

0.08

0.1 Spatial and temporal parts of the first five principal components of noise-free image time series. Essentially 100% of all variability is captured in the first four PCs, each of which is a mixture of the model functions.


RESULTS: Image Time Series MSARESULTS: Image Time Series MSA

0 20 40 60 80 100 120 140 160 180 200

-0.06

-0.04

-0.02

0

0.02

0.04

0.06

0

0.5

1

1.5

2

2.5

x 10-3

5 10 15 20 25 30 35 40

5

10

15

20

25

30

35

40

0 20 40 60 80 100 120 140 160 180 200

-0.06

-0.04

-0.02

0

0.02

0.04

0

0.5

1

1.5

2

2.5

3

3.5

x 10-3

5 10 15 20 25 30 35 40

5

10

15

20

25

30

35

40

0 20 40 60 80 100 120 140 160 180 200

-0.05

-0.04

-0.03

-0.02

-0.01

0

0.01

0.02

0.03

0.04

0.05

0

0.5

1

1.5

2

2.5

3

3.5

4x 10

-3

5 10 15 20 25 30 35 40

5

10

15

20

25

30

35

40

0

0.5

1

1.5

2

2.5

3

x 10-3

5 10 15 20 25 30 35 40

5

10

15

20

25

30

35

40

0 20 40 60 80 100 120 140 160 180 200

-0.06

-0.04

-0.02

0

0.02

0.04

MS algorithm yields essentially perfect “unmixing” of the four modeled functions, in both the spatial and temporal dimensions.


RESULTS: Detector Time Series PCA-MSARESULTS: Detector Time Series PCA-MSA

0 20 40 60 80 100 120 140 160 180 200

-0.06

-0.04

-0.02

0

0.02

0.04

0.06

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

2 4 6 8 10 12 14 16

2

4

6

8

10

12

14

16

0 20 40 60 80 100 120 140 160 180 200

-0.06

-0.04

-0.02

0

0.02

0.04

-0.6

-0.5

-0.4

-0.3

-0.2

-0.1

0

2 4 6 8 10 12 14 16

2

4

6

8

10

12

14

16

0 20 40 60 80 100 120 140 160 180 200

-0.05

-0.04

-0.03

-0.02

-0.01

0

0.01

0.02

0.03

0.04

0.05

0

0.1

0.2

0.3

0.4

0.5

2 4 6 8 10 12 14 16

2

4

6

8

10

12

14

16

0 20 40 60 80 100 120 140 160 180 200

-0.06

-0.04

-0.02

0

0.02

0.04

-0.4

-0.35

-0.3

-0.25

-0.2

-0.15

-0.1

-0.05

0

2 4 6 8 10 12 14 16

2

4

6

8

10

12

14

16

Source

Detector

Application of PCA and MSA directly to the detector time series yields four “unmixed” sets of detector data that capture essentially all of the variability and directly correspond to the four model functions.


RESULTS: Images Reconstructed from Detector MS componentsRESULTS: Images Reconstructed from Detector MS components

-2.5

-2

-1.5

-1

-0.5

0x 10


10 20 30 40

10

20

30

40

1 2 3 4-1.5

-1

-0.5

0

0.5

1Time-dependent Spatial Mean

0 1 2 3 410

-10

10-5

100

Singular Values

0 1 2 3 4100

100

100

100

100

100


-4

-3

-2

-1

0x 10


10 20 30 40

10

20

30

40

1 2 3 4-1.5

-1

-0.5

0

0.5


0 1 2 3 410

-3

10-2

10-1

100

Singular Values

0 1 2 3 498.5

99

99.5

100

100.5


0

0.5

1

1.5

2

2.5

3

x 10-3Position-dependent Temporal Mean

10 20 30 40

10

20

30

40

1 2 3 4-1

-0.5

0

0.5

1

1.5Time-dependent Spatial Mean

0 1 2 3 410

-20

10-15

10-10

10-5

100

Singular Values

0 1 2 3 4100

100

100

100

100


0

1

2

3


10 20 30 40

10

20

30

40

1 2 3 4-1

-0.5

0

0.5

1


0 1 2 3 410

-11

10-10

10-9

10-8

Singular Values

0 1 2 3 4100

100

100

100

100


Note that most ICA algorithms would be unable to distinguish these two, as they have identical histograms


RESULTS: Image Time Series GLMRESULTS: Image Time Series GLM

0

5

10

15

x 10-5

20 40

10

20

30

40

0

500

1000

1500

2000

20 40

10

20

30

40

0 500 1000-1

-0.5

0

0.5

1

0

5

10

15

x 10-5

20 40

10

20

30

40

0

1000

2000

20 40

10

20

30

40

0 500 1000-1

-0.5

0

0.5

1

0

1

2x 10

-4

20 40

10

20

30

40

0

1000

2000

3000

4000

20 40

10

20

30

40

0 500 1000-1

-0.5

0

0.5

1

0

1

2

x 10-4

20 40

10

20

30

40

0

1000

2000

3000

20 40

10

20

30

40

0 500 1000-1

-0.5

0

0.5

1

-10

-5

0x 10

-5

20 40

10

20

30

40

-1000

-500

0

20 40

10

20

30

40

0 500 10000

0.5

1

1.5

2

Lin

ear

Mod

el

Coe

ffic

ient

st-

stat

isti

c M

aps

Mod

el

Fun

ctio

ns


Noise StudyNoise Study

Modeled N/S ratio increases with increasing angle (distance) between source and detector, in agreement with usual experimental or clinical experience.


RESULTS: Images from Detector MS components, 5% noiseRESULTS: Images from Detector MS components, 5% noise

-1

0

1

2

3

4x 10


10 20 30 40

10

20

30

40

1 1.5 2 2.5 3-1

-0.5

0

0.5

1


0 1 2 310

-11

10-10

10-9

10-8

Singular Values

0 1 2 3100

100

100

100

100

100


0

1

2

3

4

5


10 20 30 40

10

20

30

40

1 1.5 2 2.5 3-1

-0.5

0

0.5

1


0 1 2 310

-9.8

10-9.6

10-9.4

10-9.2

Singular Values

0 1 2 3100

100

100

100

100


-4

-3

-2

-1

0

1


10 20 30 40

10

20

30

40

1 1.5 2 2.5 3-1.5

-1

-0.5

0

0.5


0 1 2 310

-11

10-10

10-9

10-8

Singular Values

0 1 2 3100

100

100

100

100


The same qualitative result is obtained at the 3.2% noise level; the deep inclusions merge into a single object, while the peripheral pairs remain largely isolable. When the noise level is 10%, all four dynamic model functions are overwhelmed by it.


0

1

2

x 10-4

20 40

10

20

30

40

0

5

10

15

20 40

10

20

30

40

0

0.2

0.4

0.6

0.8

20 40

10

20

30

40

0 500 1000-1

0

1

0

5

10

15

x 10-5

20 40

10

20

30

40

0

5

10

15

20 40

10

20

30

40

0

0.2

0.4

0.6

0.8

20 40

10

20

30

40

0 500 1000-1

0

1

0

10

20x 10

-5

20 40

10

20

30

40

0

20

40

60

80

20 40

10

20

30

40

0

0.2

0.4

0.6

0.8

20 40

10

20

30

40

0 500 1000-1

0

1

0

10

20x 10

-5

20 40

10

20

30

40

0

20

40

60

80

20 40

10

20

30

40

0.2

0.4

0.6

0.8

20 40

10

20

30

40

0 500 1000-1

0

1

-4

-2

0x 10

-4

20 40

10

20

30

40

-100

-50

0

20 40

10

20

30

40

0.2

0.4

0.6

0.8

20 40

10

20

30

40

0 500 10000

1

2

RESULTS: Image Time Series GLM, 3.2% NoiseRESULTS: Image Time Series GLM, 3.2% NoiseL

inea

r M

odel

C

oeff

icie

nts

t-st

atis

tic

Map

sS

igni

fica

nce

Lev

el M

aps


0

5

10

x 10-5

20 40

10

20

30

40

-2

0

2

4

20 40

10

20

30

40

0.2

0.4

0.6

0.8

20 40

10

20

30

40

0 500 1000-1

0

1

0

5

10

x 10-5

20 40

10

20

30

40

-2

0

2

4

20 40

10

20

30

40

0.2

0.4

0.6

0.8

20 40

10

20

30

40

0 500 1000-1

0

1

0

10

20

x 10-5

20 40

10

20

30

40

0

5

10

20 40

10

20

30

40

0.2

0.4

0.6

0.8

20 40

10

20

30

40

0 500 1000-1

0

1

0

10

20x 10

-5

20 40

10

20

30

40

0

5

10

20 40

10

20

30

40

0

0.2

0.4

0.6

0.8

20 40

10

20

30

40

0 500 1000-1

0

1

-10

-5

0x 10

-4

20 40

10

20

30

40

-100

-50

0

20 40

10

20

30

40

0

0.2

0.4

0.6

0.8

20 40

10

20

30

40

0 500 10000

1

2

RESULTS: Image Time Series GLM, 50% NoiseRESULTS: Image Time Series GLM, 50% NoiseL

inea

r M

odel

C

oeff

icie

nts

t-st

atis

tic

Map

sS

igni

fica

nce

Lev

el M

aps

Date post:	17-Jan-2016
Category:	Documents
Upload:	sherilyn-newman
View:	213 times
Download:	1 times

BMI2 SS07 – Class 3 “Image Processing 1” Slide 1 Biomedical Imaging 2 Class 3,4 –...

Documents