Introduction to Inverse Problems
• What is an image? Attributes and Representations • Forward vs Inverse• Optical Imaging as Inverse Problem
• Incoherent and Coherent limits• Dimensional mismatch: continuous vs discrete• Singular vs ill-posed
• Ill-posedness: a 2×2 example• Measures of ill-posedness; the Shannon metric
MIT 2.71704/11/05 – wk10-a-1
Basic premises• What you “see” or imprint on photographic film is a very narrow
interpretation of the word image• Image is a representation of a physical object having certain attributes• Examples of attributes
– Optical image: absorption, emission, scatter, color wrt light– Acoustic image: absorption, scatter wrt sound– Thermal image: temperature (black-body radiation)– Magnetic resonance image: oscillation in response to radio-
frequency EM field• Representation: a transformation upon a matrix of attribute values
– Digital image (e.g. on a computer file)– Analog image (e.g. on your retina)
MIT 2.71704/11/05 – wk10-a-2
How are images formed• Hardware
– elements that operate directly on the physical entity– e.g. lenses, gratings, prisms, etc. operate on the optical field– e.g. coils, metal shields, etc. operate on the magnetic field
• Software– algorithms that transform representations– e.g. a radio telescope measures the Fourier transform of the source
(representation #1); inverse Fourier transforming leads to a representation in the “native” object coordinates (representation #2); further processing such as iterative and nonlinear algorithms lead to a “cleaner” representation (#3).
– e.g. a stereo pair measures two aspects of a scene (representation #1); a triangulation algorithm converts that to a binocular image with depth information (representation #2).
MIT 2.71704/11/05 – wk10-a-3
Who does what• In optics,
– standard hardware elements (lenses, mirrors, prisms) perform a limited class of operations (albeit very useful ones); these operations are
• linear in field amplitude for coherent systems• linear in intensity for incoherent systems• a complicated mix for partially coherent systems
– holograms and diffractive optical elements in general perform a more general class of operations, but with the same linearity constraints as above
– nonlinear, iterative, etc. operations are best done with software components (people have used hardware for these purposes but it tends to be power inefficient, expensive, bulky, unreliable – hence these systems seldom make it to real life applications)
MIT 2.71704/11/05 – wk10-a-4
Imaging channels
HumansHumansPhysicsPhysicsHumanoidsHumanoidsAlgorithmsAlgorithms
Information generatorsInformation generators•• Wave sourcesWave sources•• Wave Wave scatterersscatterers
•• ImagingImaging•• CommunicationCommunication•• StorageStorage
UsersUsersProcessing elementsProcessing elements
GOAL:GOAL: Maximize Maximize informationinformation flowflow
MIT 2.71704/11/05 – wk10-a-5
Generalized (cognitive) representations
MIT 2.71704/11/05 – wk10-a-6
Situation ofinterest
YES/YES//NO/NO
encoded intoa scene
optical system produces a (geometricallysimilar) image
YES/YES//NO/NO
cognitiveprocessing answer
Classical “inverse problem” viewClassical “inverse problem” view--pointpoint
Situation ofinterest
YES/YES//NO/NO
encoded intoa scene
optical system produces an information-richlight intensity pattern
otherotherfunctionsfunctions
answer
““NonNon--imaging” or “generalized” sensor viewimaging” or “generalized” sensor view--pointpoint
Advantages: - optimum resource allocation- better reliability- adaptive, attentive operation
if necessary (requires resource reallocation)
e.g. is there a tanke.g. is there a tankin the scene?in the scene?
Forward problem
hardwarechannel
“physicalattributes”
(measurement)
object
fieldpropagation detection
object measurement
The Forward Problem answers the following question:• Predict the measurement given the object attributes
MIT 2.71704/11/05 – wk10-a-7
Inverse problem
hardwarechannel
“physicalattributes”
(measurement)
object
fieldpropagation detection
objectrepresentation measurement
The Inverse Problem answers the following question:• Form an object representation given the measurement
MIT 2.71704/11/05 – wk10-a-8
Optical Inversionfree space(Fresnel)
propagation
free space(Fresnel)
propagation
MIT 2.71704/11/05 – wk10-a-9
free space(Fresnel)
propagation
lens array of point-wisesensors (camera)
lensamplitude object(dark “A” on bright
background)array ofintensity
measurements
amplituderepresentation
Optical Inversion: coherent
Nonlinear problemNonlinear problem
( )yxf , ( ) ( ) ( )2
coh dd,,, ∫ −′−′=′′ yxyyxxhyxfyxIobject
amplitude intensity measurement at the output plane
Note: I could make the problem linear if I could measureamplitudes directly (e.g. at radio frequencies)
MIT 2.71704/11/05 – wk10-a-10
Optical Inversion: incoherent
Linear problemLinear problem
( )yxI ,obj ( ) ( ) ( )∫ −′−′=′′ yxyyxxhyxIyxI dd,,, incohobjmeas
objectintensity intensity measurement at the output plane
MIT 2.71704/11/05 – wk10-a-11
Dimensional mismatch• The object is a “continuous” function (amplitude or intensity)
assuming quantum mechanical effects are at sub-nanometer scales, i.e.much smaller than the scales of interest (100nm or more)– i.e. the object dimension is uncountably infinite
• The measurement is “discrete,” therefore countable and finite• To be able to create a “1-1” object representation from the
measurement, I would need to create a 1-1 map from a finite set of integers to the set of real numbers. This is of course impossible– the inverse problem is inherently ill-posed
• We can resolve this difficulty by relaxing the 1-1 requirement– therefore, we declare ourselves satisfied if we sample the object
with sufficient density (Nyquist theorem)– implicitly, we have assumed that the object lives in a finite-
dimensional space, although it “looks” like a continuous function
MIT 2.71704/11/05 – wk10-a-12
Singularity and ill-posednessUnder the finite-dimensional object assumption, the linear inverse problem
is converted from an integral equation to a matrix equation
( ) ( ) ( ) yxyyxxhyxfyxg d d , ,, −′−′=′′ ∫⇔⇔ fg H=
• If the matrix H is rectangular, the problem may be overconstrained or underconstrained• If the matrix H is square and has det(H)=0, the problem is singular; it can only be solved partially by giving up on some object dimensions (i.e. leaving them indeterminate)• If the matrix H is square and det(H) is non-zero but small, the problem may be ill-posed or unstable: it is extremely sensitive to errors in the measurement f
MIT 2.71704/11/05 – wk10-a-13
Ill-posedness: a toy problemTwo point-sources
(object)Two point-detectors
(measurement)
d1
MIT 2.71704/11/05 – wk10-a-14
A~
B~A
B
Finite-NA imaging system
A, are Gaussian conjugatesA~
B, are Gaussian conjugatesB~
Classical view
x
A~ B~
d2
z1 z2
MIT 2.71704/11/05 – wk10-a-15
Object-measurement transformation
A~ B~
ss
( ) ( ) ( )( ) ( ) ( )BAB~
BAA~
ininout
ininout
IsII
sIII
αα
αα
+=
+=
11( )
talk"-cross"
NAjincjinc 22
2
22 ⎟⎠⎞
⎜⎝⎛=⎟⎟
⎠
⎞⎜⎜⎝
⎛=
λλd
zads
( )2NA~loss
energy=α
Object-measurement transformation( ) ( ) ( )( ) ( ) ( )BAB~
BAA~
ininout
ininout
IsII
sIII
αα
αα
+=
+=
( )( )
( )( )⎟⎟⎠
⎞⎜⎜⎝
⎛⎟⎟⎠
⎞⎜⎜⎝
⎛=⎟⎟
⎠
⎞⎜⎜⎝
⎛BA
11
B~A~
in
in
out
out
II
ss
II
α
MIT 2.71704/11/05 – wk10-a-16
( )( )
( )( )
⎟⎟⎠
⎞⎜⎜⎝
⎛=
⎟⎟⎠
⎞⎜⎜⎝
⎛=⎟⎟
⎠
⎞⎜⎜⎝
⎛==
11
and ,BA
,B~A~ where,or
in
in
out
out
ss
II
II
αH
fgHfg
Hopkins matrix
measurement object
Noiseless two-point inversion
⎟⎟⎠
⎞⎜⎜⎝
⎛=
11s
sαH
( ) ⎟⎟⎠
⎞⎜⎜⎝
⎛−
−−
=−
11
11
21
ss
sαH
( ) ( )22 1det s−=αH
( ) ⎟⎟⎠
⎞⎜⎜⎝
⎛⎟⎟⎠
⎞⎜⎜⎝
⎛−
−−
=⎟⎟⎠
⎞⎜⎜⎝
⎛⇒= −
2
12
2
11
11
11
gg
ss
sff
αgHf
MIT 2.71704/11/05 – wk10-a-17
Noisy two-point inversion: ill-posedness
matrix. noise theis where,ˆLet 2
1⎟⎟⎠
⎞⎜⎜⎝
⎛=+=
nn
nnHfg
( )
( )
( ) ⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
−+−−−
==
+=+==
−
−−
221
221
1
11
1
1 iserror thewhere
, ˆˆThen
snsn
ssnn
α
αδ
δ
nHf
ffnHfHgHf
MIT 2.71704/11/05 – wk10-a-18
.1
1 asmuch asby
amplifiedisnoise so and ,1then ,0 as Note
2 ∞→−
→→
s
sd
Noisy two-point inversion:the eigenvalue/eigendirection viewpoint
MIT 2.71704/11/05 – wk10-a-19
.10
01 ,
21212121 whereor
21212121
1001
21212121
Therefore, .)1 that so d(normalize
2121 ,
2121 rseigenvecto
,1 ,1 seigenvalue has 1
1
)2()1(
)2()1(
21
⎟⎟⎠
⎞⎜⎜⎝
⎛−
+=⎟⎟
⎠
⎞⎜⎜⎝
⎛
−=∆=
⎟⎟⎠
⎞⎜⎜⎝
⎛
−⎟⎟⎠
⎞⎜⎜⎝
⎛−
+⎟⎟⎠
⎞⎜⎜⎝
⎛ −=
==
⎟⎟⎠
⎞⎜⎜⎝
⎛−=⎟⎟
⎠
⎞⎜⎜⎝
⎛=
−=+=⎟⎟⎠
⎞⎜⎜⎝
⎛=
ss
ss
sss
s
T ∆QQQH
H
ξξ
ξξ
H µµ
Noisy two-point inversion:the eigenvalue/eigendirection viewpoint
so easily), verifiedbecan (as but
as problem inverse therewrite We).1set (or moment afor Ignore
IQQQfQQQgQfQHfg
=
∆=⇒∆==
=
T
TT
αα
( ) ( )QgQgQfQfQg⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
−
+=∆=⇔∆= −
110
01
11
s
s
MIT 2.71704/11/05 – wk10-a-20
Noisy two-point inversion:the eigenvalue/eigendirection viewpoint
( )
( )
.
12
12~~̂
t,measuremennoisy theof case In the
.
2
2~ ,
2
2~Let
21
21
21
21
21
21
⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
−−+
+
+=
⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
−
+
=≡⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
−
+
=≡
snn
snn
gg
gg
ff
ff
ff
QggQff
MIT 2.71704/11/05 – wk10-a-21
Noisy two-point inversion:the eigenvalue/eigendirection viewpoint
( )
( )
( )
( )
( )( )
( )( )
.12
1 if reliable is ~̂ whereas
;12
1 if reliable is ~̂ Hence,
variance.noiseenergy if reliable ist measuremen The
.
12
12
2
2
12
12~~
~̂
~̂
2
22
22
12
2
22
22
11
21
21
21
21
21
21
2
1
2
1
−>+
+>+
>
⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
−−+
+
+⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
−
+
=
⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
−−+
+
+⎟⎟⎠
⎞⎜⎜⎝
⎛=⎟
⎟
⎠
⎞
⎜⎜
⎝
⎛
sfff
sfff
snn
snn
ff
ff
snn
snn
ff
f
f
n
n
σ
σ
MIT 2.71704/11/05 – wk10-a-22
Noisy two-point inversion:the eigenvalue/eigendirection viewpoint
( )
( )⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
−−+
+
+=
12
12~~̂21
21
snn
snn
ffionamplificat noise strong
11~~
nsuppressio noise moderate 1
1~~
2
1
−
+⇒
sf
sf
δ
δ
In the “non-resolvable” case (s→1), the “average-like quantity”n1+n2 can still be determined with moderate accuracy.
On the other hand, the “difference quantity” n1–n2 remains “swamped”by the noise. So the noise did not completely destroy the measurement,
only one component of it. In general, we can say thatill-posedness reduces the dimensionality of the measurement
(from 2→1, gradually as s→1, in this case.)MIT 2.71704/11/05 – wk10-a-23
Generalizing: ill-posednessand the square Hopkins matrix
.~E
if ~ of featureth - theresolving"" of capable is system imaging The
.~ ,~t measuremen andobject ed transform theDefine
. ,
00
0000
where, form eddiagonaliz itsin matrix Hopkins theRewrite. is and ,1 are , ,ˆ where,ˆ
2
22
)(
)2(
)1(
2
1
j
nj
mm
t
f
j
mmm
µσ
µ
µµ
>⎥⎦⎤
⎢⎣⎡
==
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
=
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
=∆
∆=
××+=
f
QggQff
ξ
ξξ
Q
QQHHfngnHfg
M
L
MOMM
L
L
MIT 2.71704/11/05 – wk10-a-24
A Hopkins matrix example
MIT 2.71704/11/05 – wk10-a-25
Imaging system
measurement
imaginglens
opticalaxis
object
• magnification = 1• no aberrations (diffraction limited)• incoherent source (i.e. point sources radiate independently)• 501 point sources, spacing d=0.05×λ/(NA)
• i.e. much denser than the “Rayleigh resolution criterion” 0.61 ×λ/(NA)
MIT 2.71704/11/05 – wk10-a-26
MIT 2.71704/11/05 – wk10-a-27
The Airy disk function
MIT 2.71704/11/05 – wk10-a-28
Intensity from incoherent sources
( )⎟⎟⎠
⎞⎜⎜⎝
⎛ −=
λNA
jinc2 dkjjkH
MIT 2.71704/11/05 – wk10-a-29
The Hopkins matrix
MIT 2.71704/11/05 – wk10-a-30
Eigenvalues of the Hopkins matrix
MIT 2.71704/11/05 – wk10-a-31
The first few significant eigenvalues
Measures of ill-posedness
Rayleigh metric; H is well-posed if R≈1,
ill-posed if R→∞min
maxRµµ
=
Shannon metric,or Image Mutual Information (IMI);
H is well-posed if C→ ∞, ill-posed if C→0.
∑=
⎟⎟⎠
⎞⎜⎜⎝
⎛+=
n
k
k
12
2
1ln21C
σµ
MIT 2.71704/11/05 – wk10-a-32
Image Mutual Information (IMI)
hardwarechannel
“physicalattributes”
(measurement)
object
fieldpropagation detection gHf
MIT 2.71704/11/05 – wk10-a-33
Assumptions: (a) f has Gaussian statistics;(b) white additive Gaussian noise (waGn)i.e. g=Hf+nwhere n is a Gaussian random vector, independent of f, with correlation matrix of the form σ2 I.
Then
quantifies information transfer between f and g.
( ) ∑=
⎟⎟⎠
⎞⎜⎜⎝
⎛+=
n
k
k
12
2
1ln21,C
σµgf H of seigenvalue :kµ
Significance of the eigenvalues
MIT 2.71704/11/05 – wk10-a-34
n
n-1
10
2nµ
21−nµ
22µ
21µ
(aka how manydimensions
the measurementis worth)
rank ofmeasurement ∑
=⎟⎟⎠
⎞⎜⎜⎝
⎛+=
n
k
k
12
2
1ln21C
σµ
...
...
largereigenvalues of H
Significance of the eigenvalues
n
n-1
10
2nµ
21−nµ
22µ
21µ
(aka how manydimensions
the measurementis worth)
rank ofmeasurement ∑
=⎟⎟⎠
⎞⎜⎜⎝
⎛+=
n
k
k
12
2
1ln21C
σµ
MIT 2.71704/11/05 – wk10-a-35
eigenvalues of H
...
larger
2σ...
Significance of the eigenvalues
n
n-1
10
2nµ
21−nµ
22µ
21µ
(aka how manydimensions
the measurementis worth)
rank ofmeasurement ∑
=⎟⎟⎠
⎞⎜⎜⎝
⎛+=
n
k
k
12
2
1ln21C
σµ
MIT 2.71704/11/05 – wk10-a-36
eigenvalues of H
...
larger
2σ
accuracyloss
...
Significance of the eigenvalues
n
n-1
10
2nµ
21−nµ
22µ
21µ
(aka how manydimensions
the measurementis worth)
rank ofmeasurement ∑
=⎟⎟⎠
⎞⎜⎜⎝
⎛+=
n
k
k
12
2
1ln21C
σµ
MIT 2.71704/11/05 – wk10-a-37
eigenvalues of H
...
larger
2σ
accuracyloss
2tµ
21−tµ ...
...
t
t-1
...
Significance of the eigenvalues
n
n-1
10
2nµ
21−nµ
22µ
21µ
(aka how manydimensions
the measurementis worth)
rank ofmeasurement ∑
=⎟⎟⎠
⎞⎜⎜⎝
⎛+=
n
k
k
12
2
1ln21C
σµ
MIT 2.71704/11/05 – wk10-a-38
eigenvalues of H larger
2σ
accuracyloss
...
...
Accuracy of the measurement
...1ln1ln1ln...
1ln21C
2
2
2
21
2
22
12
2
+⎟⎟⎠
⎞⎜⎜⎝
⎛++⎟⎟
⎠
⎞⎜⎜⎝
⎛++⎟⎟
⎠
⎞⎜⎜⎝
⎛++
=⎟⎟⎠
⎞⎜⎜⎝
⎛+=
−−
=∑
σµ
σµ
σµ
σµ
ttt
n
k
knoise floor
this term≤1
this term≈0
21
22−<< tt µσµ
≈accuracy of (t-2)th measurement
E.g. 0.5470839348
these digits worthlessif σ ≈10-5
MIT 2.71704/11/05 – wk10-a-39
Mutual information & degrees of freedom
n
n-1
10
2σ2nµ
21−nµ
22µ
21µ
rank ofmeasurement
∑=
⎟⎟⎠
⎞⎜⎜⎝
⎛+=
n
k
k
12
2
1ln21C
σµ
...
...
MIT 2.71704/11/05 – wk10-a-40
2σ
mutualinformation
As noise increases• one rank of H is lost wheneverσ2 overcomes a new eigenvalue• the remaining ranks lose precision
IMI for two-point resolution problem
11
2
1
−=+=
ss
µµ
⎟⎟⎠
⎞⎜⎜⎝
⎛=
11s
sH ( ) 21det s−=H
⎟⎟⎠
⎞⎜⎜⎝
⎛−
−−
=−
11
11
21
ss
sH
( ) ( ) ( )⎟⎟⎠
⎞⎜⎜⎝
⎛ +++⎟⎟
⎠
⎞⎜⎜⎝
⎛ −+= 2
2
2
2 11ln2111ln
21,C
σσssGF
MIT 2.71704/11/05 – wk10-a-41
MIT 2.71704/11/05 – wk10-a-42
IMI vs source separation
( ) 2
1SNRσ
=
s → 0s → 1
Concluding remarks
MIT 2.71704/11/05 – wk10-a-43
• “Exceeding the resolution limit” does not imply a catastrophic degradation in the ability of the system to image; it merely implies gradual loss of accuracy and, possibly, reduction in the number of “degrees of freedom” (i.e., the dimensionality) of the measurement.
• The losses of accuracy and degrees of freedom increase monotonically (i.e., become worse) with the level of measurement noise.
• The Shannon metric conveniently quantifies the gradual losses ofaccuracy and degrees of freedom in terms of the relative magnitude of the eigenvalues of the Hopkins matrix and the noise variance.
• Our formula for the Shannon metric is based on the assumption of both object and noise being Gaussian, incoherent, and completely uncorrelated to each other; it turns out this is a worst case assumption. Under more general conditions, the Shannon metric usually must be computed numerically (e.g., Monte-Carlo simulation.)
• The Shannon metric expresses an upper limit in the worst-case performance of the imaging system for the given Hopkins matrix, but does not specify the “inversion algorithm” that should be used to achieve this limit.