Linear Inverse Problems
A MATLAB TutorialPresented by Johnny Samuels
What do we want to do?
• We want to develop a method to determine the best fit to a set of data: e.g.
The Plan
• Review pertinent linear algebra topics• Forward/inverse problem for linear systems• Discuss well-posedness• Formulate a least squares solution for an
overdetermined system
Linear Algebra Review• Represent m linear equations with n variables:
• A = m x n matrix, y = n x 1 vector, b = m x 1 vector
• If A = m x n and B = n x p then AB = m x p (number of columns of A = number of rows of B)
11 1 12 2 1 1
21 1 22 2 2 2
1 1 2 2
n n
n n
m m mn n m
a y a y a y ba y a y a y b
a y a y a y b
1 111 1
1
n
m mn n m
y bA
y ba a
a a y b
Linear Algebra Review:Example
•
• Solve system using MATLAB’s backslash operator “\”: A = [1 -1 3 1;3 -3 1 0;1 1 0 -2]; b=[2;-1;3]; y = A\b
1 2 3 4
1 2 3
1 2 4
3 23 3 1
2 2
y y y yy y yy y y
1
2
3
4
1 1 3 1 23 3 1 0 11 1 0 2 3
bAy
yyyy
Linear Algebra Review:What does it mean?
•
• Graphical Representation:
1 2
1 2
12 1
y yy y
1
2
1 1 12 1 1
byA
yy
Linear Algebra Review:Square Matrices
• A = square matrix if A has n rows and n columns
• The n x n identity matrix =
• If there exists s.t. then A is invertible
1 0 0 00 1 0 0
0 1
I
1A 1A A I
Linear Algebra ReviewSquare Matrices cont.
• If then
• Compute (by hand) and verify (with MATLAB’s “*” command) the inverse of
a bA
c d
1 1 d bA
c aad bc
2 11 3
A
Linear Algebra Review:One last thing…
• The transpose of a matrix A with entries Aij is defined as Aji and is denoted as AT – that is, the columns of AT are the rows of A
• Ex: implies
• Use MATLAB’s “ ‘ “ to compute transpose
1 2 30 0 4
A
1 02 03 4
TA
Forward Problem:An Introduction
• We will work with the linear system Ay = b where (for now) A = n x n matrix, y = n x 1 vector, b = n x 1 vector
• The forward problem consists of finding b given a particular y
Forward Problem:Example
• g = 2y : Forward problem consists of finding g for a given y
• If y = 2 then g = 4
• What if and ?
• What is the forward problem for vibrating beam?
47 2889 53
A
11
y
Inverse Problem
• For the vibrating beam, we are given data (done in lab) and we must determine m, c and k.
• In the case of linear system Ay=b, we are provided with A and b, but must determine y
Inverse Problem:Example
• g = 2y : Inverse problem consists of finding y for a given g
• If g = 10 then
•
•
• Use A\b to determine y
1 12 2 2 10 5y y
1
2
3
0 1 1 32 4 1 12 5 4 2
byA
yyy
1
1
2
3
0 1 1 32 4 1 12 5 4 2
yyy
1 1 1Ay b A Ay A b y A b
Well-posedness• The solution technique produces
the correct answer when Ay=b is well-posed
• Ay=b is well-posed when 1. Existence – For each b there exists an y such that
Ay=b2. Uniqueness – 3. Stability – is continuous
• Ay=b is ill-posed if it is not well-posed
1y A b
1 2 1 2Ay Ay y y 1A
Well-posedness:Example
• In command window type y=well_posed_ex(4,0)
• y is the solution to
• K = condition number; the closer K is to 1 the more trusted the solution is
1
2
3
4
1 0 0 0 10 1 0 0 10 0 1 0 10 0 0 1 1
byA
yyyy
Ill-posedness:Example
• In command window type y=ill_posed_ex(4,0)
• y is the solution to where
• Examine error of y=ill_posed_ex(8,0)
• Error is present because H is ill-conditioned
1111
H y
1.0000 0.5000 0.3333 0.2500 0.5000 0.3333 0.2500 0.2000 0.3333 0.2500 0.2000 0.1667 0.2500 0.2000 0.1667 0.1429
H
What is an ill conditioned system?
• A system is ill conditioned if some small perturbation in the system causes a relatively large change in the exact solution
• Ill-conditioned system:
Ill-Conditioned System:Example II
•
•
1
2
.835 .667 .168
.333 .266 .067yA b
yy
1
2
??
yy
1
2
.835 .667 .168
.333 .266 .066yA b
yy
1
2
??
yy
What is the effect of noisy data?
• Data from vibrating beam will be corrupted by noise (e.g. measurement error)
• Compare:1. y=well_posed_ex(4,0) and z=well_posed(4,.1)2. y=well_posed_ex(10,0) and z=well_posed(10,.2)3. y=ill_posed_ex(4,0) and z=ill_posed(4,.1)4. y=ill_posed_ex(10,0) and z=ill_posed(10,.2)
• How to deal with error? Stay tuned for next talk
Are we done?
• What if A is not a square matrix? In this case does not exist
• Focus on an overdetermined system (i.e. A is m x n where m > n)
• Usually there exists no exact solution to Ay=b when A is overdetermined
• In vibrating beam example, # of data points will be much larger than # of parameters to solve (i.e. m > n)
1A
Overdetermined System:Example
•
•
• Minimize
22
TAy b Ay b Ay b
2 2 2 21 22 1
n
i ni
x x x x x
Obtaining the Normal Equations• We want to minimize :
• is minimized when y solves• provides the least squares
solution
Ty Ay b Ay b
2
TTT
T T
T T T T
T T
y A Ay b Ay b A
A Ay b A Ay b
A Ay A b A Ay A b
A Ay A b
y T TA Ay A b
1T Ty A A A b
Least Squares:Example
• Approximate the spring constant k for Hooke’s Law: l is measured lengths of spring, E is equilibrium position, and F is the resisting force
• least_squares_ex.m determines the least squares solution to the above equation for a given data set
byA
l E k F
1T Tk l E l E l E F
What did we learn?
• Harmonic oscillator is a nonlinear system, so the normal equations are not directly applicable
• Many numerical methods approximate the nonlinear system with a linear system, and then apply the types of results we obtained here