Yet another mathematical framework for understanding DSR Contents Motivation................................................ 2 Assumptions............................................... 2 Image observation model...................................2 Objective: recover image sensed by detector with smaller pixels.................................................... 3 Image observation model (continued…)......................4 Influence of “detector” shifts on the detected image......9 Influence of “object” shifts on the detected image.......10 Exact recovery of the optical image in noiseless DSR: Fact or fiction?.............................................. 13 Derivation of the 1D Generalized Sampling Theorem (GST). .14 Connection between GST and Digital Super Resolution......17 Digital Super Resolution - Reduction to practice.........20 TODO..................................................... 20 Other observations.......................................20 Lingering Questions......................................20 Appendix – Useful identities.............................21 Accommodating fill factors ¿ 100 % .........................22 Figures Figure 1 Relationship between geometric image of scene and detected image............................................3 Figure 2 Relationship between geometric image of scene and desired image............................................. 3
Transcript
Yet another mathematical framework for understanding DSR
Accommodating fill factors ¿100 %..................................................................................................22
FiguresFigure 1 Relationship between geometric image of scene and detected image.............3Figure 2 Relationship between geometric image of scene and desired image................3Figure 3 Relationship between the geometric image of scene and the desired, detected images........................................................................................................................................... 8Figure 4 Generalized sampling configuration.............................................................................14Figure 5 Interpreting Digital Super Resolution as a variant of the standard GST configuration.............................................................................................................................................. 17
MotivationThe vast body of literature on Digital Super Resolution fails to provide a definite answer to fundamental questions such as
Is perfect recovery of an optically band-limited image possible under precise knowledge of translational motion, in the absence of noise?
What are preferred optical PSF’s and focal plane masks for digital super-resolution?
Are there super-resolution schemes that are free of regularization? What are necessary and sufficient conditions for super-resolution? What is the connection between Digital Super Resolution and Papoulis’s
Generalized Sampling Theorem? What is the connection between Digital Super Resolution and a maximally
decimated perfect reconstruction filter-bank? The present work tries to answer these questions by developing a mathematical framework for understanding Digital Super Resolution.
Assumptions1. Circular aperture and space-invariant optical PSF2. The inter-sample-spacing of the sampled optical PSF given by Δhi exceeds the
optical cutoff frequency2NAλ where NA is the numerical aperture, and λ is
the wavelength of light expressed in meters. 3. An integer number of higher-resolution pixels (size Δhi) can be perfectly
accommodated within a single detector pixel (size Δlo). This implies that G=Δhi÷ Δlo∈Z+¿¿
.4. The pixels are square in shape, and have 100 % fill factor5. The central pixel in the detector is assumed to be the origin of the image
coordinate system.
Image observation modelilo [m,n]=∬ [o ( x , y )⊛hopt ( x , y ) ]hdet (x−m Δlo , y−n Δlo)dxdy (1)
o (x , y ) geometric image of the scene from the camera vantage point
hopt (x , y ) intensity PSF of the optics(accommodates blurring induced by the optics)
Δlo Size of individual pixel in detector
hdet (x−m Δlo , y−n Δlo)=rect ( x−m Δlo
Δlo)rect ( y−n Δlo
Δlo)spatial response of (m ,n )th light sensitive
element in the detector.(accommodates blurring induced by pixel geometry)
Figure 1 Relationship between geometric image of scene and detected image
H opt (ξ , η)≝F {hopt ( x , y ) } optical transfer function
Δ lo2 sinc (−Δloξ ) sinc (−Δlo η )=Δlo
2 sinc (Δ loξ ) sinc ( Δlo η )low resolution detector pixel transfer function F {hdet ( x , y ) }
Objective: recover image sensed by detector with smaller pixelsTo be precise, we would like to estimate samples o [k , l ] from the detected observations ilo [m,n]. To this extent, it helps to know the expression for o [k , l ]
o [k , l ]≝∬ {o (x , y )⊛sinc( 2NAλ √x2+ y2)} rect ( x−k Δhi
Δhi )rect ( y−l Δhi
Δhi )dxdy={o ( x , y )⊛sinc( 2NAλ √x2+ y2)⊛[ rect(−x
circ( λ2NA √ξ2+η2) transfer function of rotationally symmetric
ideal low pass filter
NotesThe following paragraphs outline a method for recovering samples o [k , l ] obtained at intervals of Δhi, using a detector with pixel pitch Δlo=G Δhi , s.tG∈Z+¿¿.
Image observation model (continued…)With a little effort, (1) can be recast in the following form
ilo [m ,n ]={o ( x , y )⊛ [hopt (x , y )⋆hdet (x , y ) ] }δ (x−mΔlo , y−n Δlo )={o ( x , y )⊛ [hopt ( x , y )⋆hdet ( x , y ) ] }δ (x−mG Δhi , y−nG Δhi )(3)
The second expression in (3) follows from the assumption that an integer number of higher-resolution pixels can be perfectly accommodated within a single detector pixel.
Suppose that the optical PSF is band-limited such that its bandwidth does not exceed 0.5÷ Δhi. In such a case, the 2D extension of the sampling theorem [Pg.88 of A.K. Jain, Oppenheim] may be used to express the optical PSF as follows
hopt ( x , y )= ∑i , j∈ Z
h [i , j ] sinc( x−i Δhi
Δhi)sinc( y− j Δhi
Δhi) (4)
The entries h [ i , j ] are obtained by directly sampling the optical PSF at intervals of Δhi
, as shown below
h [ i , j ]≝hopt (i Δhi , j Δhi ) (5)
The constraint that Δlo=G Δhi , s.tG∈Z+¿¿ implies that an integer number of higher-resolution pixels (square in shape) can be perfectly accommodated within a single detector pixel (also square in shape). Consequently,
hdet (x , y )= ∑k , l∈Z
d [k ,l ] rect ( x−k Δhi
Δhi)rect ( y−l Δhi
Δhi) (6)
The weights d [k ,l ] are defined as follows
d [k ,l ]≝∬hdet ( x , y ) rect ( x−k Δhi
Δhi)rect( y−l Δhi
Δhi)dxdy (7)
In its present form, (6) suggests that the array d [k ,l ] has infinite spatial extent. But, in practice it has finite spatial extent and behaves as a Finite-Impulse-Response
filter. The spatial response function hdet (x , y ) is expressed as an infinite sum for the sake of mathematical convenience.
Typically, the non-zero components of d [k ,l ] are identical, corresponding to a summing filter. But, the weights d [k ,l ] maybe interpreted in a broader context as being derived from a pixelated focal-plane-mask (such as a Hadamard mask) with a feature size of Δhi.
Using (4), (6) we will now attempt to evaluate the term hopt ( x , y )⋆hdet ( x , y ) in (3).
hopt ( x , y )⋆hdet ( x , y )=[( ∑i , j∈Z h [ i , j ]δ ( x−i Δhi , y− j Δhi ))⊛(sinc ( xΔhi )sinc( y
Δhi ))]⋆[( ∑k ,l∈Z d [k , l ] δ (x−k Δhi , y−l Δhi ))⊛(rect ( xΔhi )rect ( y
Δhi ))](8)
Further simplification is possible by utilizing a special property of the convolution and correlation for real signals
[ s1(x , y )⊛s2( x , y) ]⋆ [s3 ( x , y )⊛ s4 ( x , y ) ]=[ s1 (x , y )⋆ s3 ( x , y ) ]⊛ [ s2 ( x , y )⋆ s4 ( x , y ) ] (9)
A proof of the above property is included in the Appendix. In view of the property, one may simplify (10) as follows
hopt ( x , y )⋆hdet ( x , y )=[( ∑i , j∈Z h [ i , j ]δ (x−i Δhi , y− j Δhi ))⋆( ∑k ,l∈Z d [k , l ]δ (x−k Δhi , y−l Δhi ))]⊛ [(sinc( xΔhi )sinc( y
Δhi ))⋆(rect ( xΔhi ) rect( y
Δhi ))](10)
Additional simplification is possible by utilizing a It follows from the definition of the Dirac-delta function that
In view of the above property, one may simplify (10) as shown below
hopt ( x , y )⋆hdet ( x , y )=[ ∑i , j , k ,l∈Z
d [k , l ]h [ i , j ]δ ( x−(k−i )Δhi , y−(l− j )Δhi )]⊛[(sinc( xΔhi )sinc ( y
Δhi ))⋆(rect ( xΔhi )rect ( y
Δhi ))](12)
Under the change of variables~i=k−i ,~j=l− j, the left operand of the convolution may be expressed as the cross-correlation of the discrete arraysd [… ]∧h [… ], so that
hopt ( x , y )⋆hdet ( x , y )=[ ∑~i ,~j∈Z { ∑k , l∈Z d [k ,l ]h [k−~i , l−~j ]}δ (x−~i Δhi , y−~j Δhi )]⊛ [(sinc( x
Δhi )sinc( yΔhi ))⋆(rect( x
Δhi )rect ( yΔhi ))]=[ ∑~i ,~j∈ Z
Γdh [ ~i ,~j ] sinc( x−~i Δhi
Δhi)sinc( y−
~j Δhi
Δhi)]⏟
heff ( x , y )
⋆[rect ( xΔhi) rect( y
Δhi )](13)
The term heff (x , y ) may be interpreted as the expression for a band-limited image that is reconstructed from uniformly spaced samples with an inter-sample-spacing of Δhi. In subsequent expressions, the aforementioned term will be denoted as the effective PSFheff (x , y ).
Substituting (13) in (1) yields the following expression for the intensity of the (m ,n )th pixel of the detected image
ilo [m ,n ]={o ( x , y )⋆[rect ( xΔhi ) rect ( y
Δhi) ]⊛heff (x , y )}δ (x−mG Δhi , y−nG Δhi)={o ( x , y )⊛[rect (−xΔhi )rect (− y
Δhi )]⊛heff ( x , y)}δ (x−mG Δhi , y−nG Δhi )(14)
Diffraction limits the largest spatial frequency that can be resolved by the optics to λNA
≤ 1Δhi
, where λis the wavelength of illumination, and NA is the numerical
aperture.Consequently, post-factum filtering of heff ( x , y )=hopt ( x , y )⋆hdet ( x , y ) by an ideal low-
pass filter such as sinc( 2NAλ √ x2+ y2) has no effect onheff ( x , y ). In other words,
heff ( x , y )⊛sinc( 2NAλ √ x2+ y2)=heff (x , y ) . In view of this observation, one can rewrite
(14) as follows
ilo [m ,n ]={o ( x , y )⊛[rect (−xΔhi )rect (− y
Δhi )]⊛
sinc( 2NAλ √x2+ y2)⊛heff (x , y )}δ (x−mG Δhi , y−nGΔhi )={~o ( x , y )⊛heff (x , y )}δ (x−mG Δhi , y−nG Δhi )(15)
The definition of the term ~o (x , y ) in (2) is utilized to derive the final expression in (15).
According to the Shannon-Whittaker sampling theorem, the band-limited image ~o (x , y ) admits the following expansion
~o (x , y )= ∑k , l∈ Z
o [k , l ] sinc( x−k Δhi
Δhi)sinc( y−l Δhi
Δhi)where o [k , l ]=~o (k Δhi , l Δhi) (16)
We know from (13) that the effective PSF heff (x , y ) admits a similar series expansion. By combining the identities (13) & (16), one can determine the expression for the image~o (x , y )⊛heff (x , y ), which when sampled at regular intervals yields the intensity of the (m ,n )th detector pixel. The result is included below
~o (x , y )⊛heff (x , y )=( ∑k ,l∈Z o [k ,l ] sinc( x−k Δhi
Δhi )sinc( y−l Δhi
Δhi ))⊛( ∑i , j∈ ZΓdh [i , j ] sinc( x−i Δhi
Δhi )sinc( y− j Δhi
Δhi ))= ∑i , j ,k ,l∈ Z
o [k , l ] Γ dh [ i , j ] sinc( x−(i+k )Δhi
Δhi )sinc( y−( j+l)Δhi
Δhi )(17)
In deriving (17), the property sinc( xΔhi )⊛sinc( x
Δhi )=sinc( xΔhi ) was utilized.
With the aid of (17), one can deduce the intensity of the (m ,n )th detector pixel. The result is included below
ilo [m ,n ]={~o (x , y )⊛heff (x , y )}δ (x−mG Δhi , y−nG Δhi )= ∑i , j , k ,l∈Z
o [k ,l ] Γdh [mG−k ,nG−l ]={o [k ,l ]⊛Γ dh [k , l ] }↓G(18)
In deriving (18), the following property of the sinc(…) function is utilized
for i , j∈Z+¿ ,sinc (i , j)=( sin ( πi )
πi )( sin (πj )πj )={ 1 ,∧i= j=0
0 ,∧otherwise¿ (19)
In summary, the intensity of the (m ,n )th pixel may be obtained as follows 1. uniformly sample the band-limited image
~o (x , y )=o ( x , y )⊛ rect(−xΔhi
,− yΔhi )⊛sinc( 2NA
λ √x2+ y2) at intervals of width Δhi,
to obtain the image o [k , l ]2. convolve the sampled image with the array Γdh [k ,l ]=d [k ,l ]⋆h[k , l ] where
h [k , l ] is the sampled optical PSF, and d [k , l ] is the sampled detector spatial response
3. down-sample the result by a factor of G
≡
Figure 3 Relationship between the geometric image of scene and the desired, detected images
H (ξ , η) Discrete Time Fourier Transform of the sampled optical PSF h [k , l ]≝hopt(k Δhi , l Δhi)
D (ξ , η ) Discrete Time Fourier Transform of
d [k ,l ]≝∬hdet ( x , y ) rect ( x−k Δhi
Δhi)rect ( y−l Δhi
Δhi )dxdyFacts In the absence of a focal plane mask
hdet (x , y )=rect ( xΔlo )rect ( y
Δlo ). The array d [k ,l ]has exactly G2 non-zero entries when G is an
integer. These entries are 1 in the absence of a focal plane mask, and correspond to a G×G code in the presence of a focal plane mask
Influence of “detector” shifts on the detected imageFor the purpose of this analysis, we assume that the detector experiences a translational motion of ( p Δhi ,q Δhi)meters, prior to integration. It will be assumed that s.t p ,q∈Z+¿ .¿
By definition,
ilo [m ,n; p ,q ]≝∬ [ o (x , y )⊛hopt ( x , y ) ]hdet ((x−p Δhi)−m Δlo , ( y−q Δhi )−n Δlo )dxdy= {o ( x , y )⊛ [hopt ( x , y )⋆hdet (x− p Δhi , y−q Δhi ) ]}δ (x−Δlo , y−Δlo)={o ( x , y )⊛ [hopt ( x , y )⋆hdet (x−p Δhi , y−q Δhi) ]}δ (x−mGΔhi , y−nG Δhi)(20)
In an effort to evaluate (20) we recall the expression for hopt ( x , y )⋆hdet ( x , y ) listed in (13),
hopt ( x , y )⋆hdet ( x , y )=[ ∑i , j∈Z Γdh [ i , j ] sinc( x−i Δhi
Δhi )sinc( y− j Δhi
Δhi )]⏟heff (x , y)
⋆[rect( xΔhi )rect ( y
Δhi )](21)
The definition of the various terms referenced in (21) is included below
h [ i , j ]≝hopt (i Δhi , j Δhi )
d [k ,l ]≝∬hdet ( x , y ) rect ( x−k Δhi
Δhi)rect( y−l Δhi
Δhi)dxdy
Γdh [ i , j ]≝ ∑k , l∈Z
d [k ,l ]h [k−i , l− j ]
(22)
We know from the discussion in the previous section that the effective PSF heff ( x , y ) is unaffected by convolution with an ideal low pass filter, i.e.
heff ( x , y )⊛sinc( 2NAλ √ x2+ y2)=heff (x , y ) (23)
Incorporating (21), (22) and (23) into (20) yields the following expression for the intensity ilo [m ,n; p ,q ]
ilo [m ,n; p ,q ]={~o ( x , y )⊛heff (x−p Δhi , y−q Δhi)}δ (x−mG Δhi , y−nG Δhi ) where
~o (x , y )≝ o (x , y )⊛ [rect (−xΔhi )rect(− y
Δhi ) ]⊛sinc( 2NAλ √x2+ y2)⊛heff (x , y)
(24)
Following (17), we can express ~o (x , y )⊛heff (x−p Δhi , y−q Δhi) as follows
~o (x , y )⊛heff (x−p Δhi , y−q Δhi)=( ∑k, l∈Z o [k , l ] sinc( x−k Δhi
Incorporating (25) into (20) yields the following expression for the intensity ilo [m ,n; p ,q ]
ilo [m ,n; p ,q ]={~o ( x , y )⊛heff (x−p Δhi , y−q Δhi ) }δ (x−mG Δhi , y−(nG+q)Δhi )= ∑i , j ,k , l∈Z
o [k , l ] Γ dh [ i , j ] sinc ( (mG−k−p )−i ) sinc ( (mG−l−q )− j )= ∑k ,l∈ Z
o [k , l ] Γ dh [(mG−p)−k ,(nG−q)−l ]= {o [k , l ]⊛Γdh [k−p , l−q ] }↓G(26)
In deriving (25), the following properties of the sinc(…) function is utilized
sinc( xΔhi )⊛sinc( x
Δhi )=sinc( xΔhi )
for i , j∈Z+¿ ,sinc (i , j)=( sin (πi )
πi )( sin (πj )πj )={ 1 ,∧i= j=0
0 ,∧otherwise¿
(27)
In summary, the intensity of the (m ,n )th low-res image pixel for a detector translation of ( p Δhi ,q Δhi)meters, may be computed as follows
1. uniformly sample the band-limited image
~o (x , y )=o ( x , y )⊛ rect(−xΔhi
,− yΔhi )⊛sinc( 2NA
λ √x2+ y2) at intervals of width Δhi,
to obtain the image o [k , l ]2. convolve the sampled image with the array Γdh [k−p , l−q ]=d [k , l ]⋆h [k ,l ]
where h [k , l ] is the sampled optical PSF, and d [k , l ] is the sampled detector spatial response
3. down-sample the result by a factor of G
Influence of “object” shifts on the detected imageFor the purpose of this analysis, we assume that the object experiences translational motion of ( p Δhi ,q Δhi)meters, prior to imaging. It will be assumed that s.t p ,q∈Z+¿ .¿
By definition,
ilo [m ,n; p ,q ]≝∬ [o (x−p Δhi , y−q Δhi )⊛hopt ( x , y ) ]hdet (x−mΔlo , ( y−q Δhi )−n Δlo )dxdy= {o (x− p Δhi , y−q Δhi )⊛ [hopt ( x , y )⋆hdet ( x , y ) ] }δ ( x−Δlo , y−Δlo )= {o (x− p Δhi , y−q Δhi )⊛ [hopt ( x , y )⋆hdet ( x , y ) ] }δ ( x−mGΔhi , y−nG Δhi )(28)
In an effort to evaluate (20) we recall the expression for hopt ( x , y )⋆hdet ( x , y ) listed in (13),
hopt ( x , y )⋆hdet ( x , y )=[ ∑i , j∈Z Γdh [ i , j ] sinc( x−i Δhi
Δhi )sinc( y− j Δhi
Δhi )]⏟heff (x , y)
⋆[rect( xΔhi )rect ( y
Δhi )](29)
The definition of the various terms referenced in (21) is included below
h [ i , j ]≝hopt (i Δhi , j Δhi )
d [k ,l ]≝∬hdet ( x , y ) rect ( x−k Δhi
Δhi)rect( y−l Δhi
Δhi)dxdy
Γdh [ i , j ]≝ ∑k , l∈Z
d [k ,l ]h [k−i , l− j ]
(30)
We know from the discussion in the previous section that the effective PSF heff ( x , y ) is unaffected by convolution with an ideal low pass filter, i.e.
heff ( x , y )⊛sinc( 2NAλ √ x2+ y2)=heff (x , y ) (31)
Incorporating (21), (22) and (23) into (20) yields the following expression for the intensity ilo [m ,n; p ,q ]
ilo [m ,n; p ,q ]={~o (x−p Δhi , y−q Δhi )⊛heff (x , y )}δ (x−mG Δhi , y−nG Δhi ) where
~o (x , y )≝ o (x , y )⊛ [rect (−xΔhi )rect(− y
Δhi ) ]⊛sinc( 2NAλ √x2+ y2)⊛heff (x , y)
(32)
Following (17), we can express ~o ( x−p Δhi , y−q Δhi)⊛heff (x , y ) as follows
o [k , l ] Γ dh [ i , j ] sinc ( (mG−k−p )−i ) sinc ( (mG−l−q )− j )= ∑k , l∈ Z
o [k , l ] Γ dh [(mG−p)−k ,(nG−q)−l ]= {o [k , l ]⊛Γdh [k−p , l−q ] }↓G(34)
In deriving (34), the following properties of the sinc(…) function is utilized
sinc( xΔhi )⊛sinc( x
Δhi )=sinc( xΔhi )
for i , j∈Z+¿ ,sinc (i , j)=( sin (πi )
πi )( sin (πj )πj )={ 1 ,∧i= j=0
0 ,∧otherwise¿
(35)
In summary, the intensity of the (m ,n )th low-res image pixel for a detector translation of ( p Δhi ,q Δhi)meters, may be computed as follows
1. uniformly sample the band-limited image
~o (x , y )=o ( x , y )⊛ rect(−xΔhi
,− yΔhi )⊛sinc( 2NA
λ √x2+ y2) at intervals of width Δhi,
to obtain the image o [k , l ]2. convolve the sampled image with the array Γdh [k−p , l−q ]=d [k , l ]⋆h [k ,l ]
where h [k , l ] is the sampled optical PSF, and d [k , l ] is the sampled detector spatial response
3. down-sample the result by a factor of G
Exact recovery of the optical image in noiseless DSR: Fact or fiction?Optimization problem we wish to solve ??
Need to work on this section
Key points One can reconstruct o [k , l ] in the absence of noise only iff
o There are no nulls in the DTFT of the sampled optical PSF h [k , l ]o There are no nulls in the DTFT of the sampled detector response
d [k , l ] In mathematical terms, this implies that H (ξ , η )≠0 and D (ξ ,η )≠0 for
ξ ,η∈( −12 Δhi
, 12 Δhi ).
Failure to meet these constraints result in an irreversible loss of the frequencies. This suggests that clear pixels with large footprint may be not be well suited for perfect recovery of the optical image.ExampleSupposeG=4. Consider a two-valued high-res spatial pattern o [k , l ] that has intensities I 1 in columns 1,2,5,6,9,10… and intensities I 2 in columns
3,4,7,8,11,12….The frequency of this pattern is 1
4 Δhi= 4
4 Δlo= 1Δlo
Sub-pixel shifting followed by integration over 4×4 neighborhood always yields the same value, so that ilo [m ,n; p ,q ] is always gray. No amount of regularization can help recover this frequency. The most appropriate prior for reconstruction is a sparsity prior for the object, but even that will not work as a constant gray image is the sparsest solution to the regularization problem. QuestionIs there a non-uniform spatial pattern o [k , l ] such that ilo [m ,n; p ,q ]=fixed value for all0≤ p ,q<G. If so, will any reconstruction algorithm claim that o [k , l ] is constant over a G×G neighborhood? I think the answer is yes, and these patterns correspond to nulls inD (ξ ,η ).
Identify necessary and sufficient condition for exact recovery as
Derivation of the 1D Generalized Sampling Theorem (GST)Claim: A band-limited function f (t) is uniquely determined in terms of samples gk (nT ) of the responses gk (t ) of m linear systems with inputf (t), sampled at 1/m of the Nyquist rate.
Figure 4 Generalized sampling configuration
CommentsThe GST of Papoulis deals with the configuration shown in Figure 4, wherein a single band-limited signal f (t) is fed into M linear time-invariant filters (pre-filters), and each filter output is then sampled at ¿ the Nyquist rate. Under certain conditions on the pre-filters and in the absence of noise, the input can be reconstructed “exactly” from the samples acquired by sampling the filter outputs.
NotesThe following derivation borrows heavily and shamelessly from the following sources:
1. “Generalized sampling expansion” by A. Papoulis2. “Multichannel sampling of low-pass signals” by John L. Brown Jr.3. “Classical sampling theorems in the context of multirate and polyphase
digital filter bank structures” by P.P. Vaidyanathan and V.C. Liu
ProofFor the purpose of this proof, it is assumed that the signal f (t) is band-limited and square integrable, such that its spectrum F (ν ) vanishes outside the interval
(−0.5T 0
, 0.5T 0 ), whereT 0 is the sampling inerval. In subsequent discussions, the
sampling rate 1T0
will be denoted by the symbol νs .
The proof begins by identifying the expression for the sampled output of the k th filter in the bank of pre-filters,
The spectrum of the k th sampled signal gk (t ) is given by
Gk (ν )=F {gk ( t ) }=[F ( ν )H k ( ν ) ]⊛ [ νsM ∑n'∈ Z
δ (ν−n 'ν sM )]= νs
M ∑n'∈ Z
F (ν−n 'νsM )H k (ν−n '
ν sM )(37)
Following a change of variables(n=−n' ), (37) may be rewritten as shown below
Gk (ν )=F {gk (t ) }=νsM ∑
n∈ZF (ν+n νs
M )H k (ω+nωs
M ) (38)
The sampled signals {gk ( t ) }k=1M
serve as input to the reconstruction filters{ y k (t ) }k=1M
. The
expression for the reconstructed signal ~f (t ) is given below
~f (t)=∑m=1
M
gk ( t )⊛ yk ( t ) (39)
The spectrum of the reconstructed signal ~f (t)is given by
~F (ν )=∑m=0
M−1
F {gk (t )⊛ yk (t ) }=∑m=0
M−1
Gk (ν )Y k (ν)=νsM ∑
k=0
M−1 [∑n∈Z F (ν+n νsM )H k (ν+n νsM )]Y k (ν)=νsM ∑
n∈ZF (ν+n ν sM )[∑k=1
M
H k(ν+n νsM )Y k (ν )](40)
Perfect recovery of the signal f (t) demands that~F (ν )=F(ν). But then, (40) in its present form provides little insight into the existence of M analysis and synthesis filters {hk ( t ) , yk (t ) }k=1
M such that~F (ν )=F(ν).
In an attempt to address the issue, we gather terms in the outer summation of (40) whose indices n yield a reminder of 0…M−1 when divided byM , i.e.
~F (ν )=ν sM ∑
i=0
M−1 { ∑n∈Z
mod (n ,M )=i
F (ν+n νsM )[∑k=1
M
H k (ν+n ν sM )Y k (ν)]} (41)
The expression for ~F (ν ) provided in (41) may be simplified by utilizing the one-to-one correspondence between elements of the set{n∈Z s.tmod (n , M )=i} and the set {l∈Z s.t n=i+M l }.
The simplified expression for ~F (ν ) is shown below
~F (ν )=ν sM ∑
i=0
M−1 {∑l∈ ZF(ν+(i+M l)
ν sM ) [∑k=1
M
H k(ν+(i+M l)νsM )Y k (ν )]}= ν s
M ∑i=0
M−1 {∑l∈ ZF((ν+i νsM )+ l νs)[∑k=1
M
H k ((ν+i νsM )+l νs)Y k (ν )]}(42)
Additional simplification is possible by recognizing that the passband of the reconstruction filters Y k (ν) need only be confined to the interval(−0.5 νs ,0.5νs ), given that the spectrum of the input signal f (t ) vanishes outside the interval(−0.5 νs ,0.5νs ).
Please note that the above restriction is in line with the established practice of reconstructing a band-limited signal from its un-aliased samples, by employing an ideal LPF whose cutoff frequency matches the Nyquist frequency.
Incorporating the above constraint in the expression for ~F (ν ) amounts to discarding terms that correspond tol ≠0. The final expression is shown below
~F (ν )=ν sM ∑
i=0
M−1 {F (ν+ i νsM )[∑k=1
M
H k (ν+i ν sM )Y k (ν)]} (43)
Careful inspection of (43) yields a set of constraints must be simultaneously satisfied for perfect recovery, via~F (ν )=F(ν). These include
∑k=1
M
H k ( ν )Y k ( ν )=M /νs
∑k=1
M
H k (ν+ i νsM )Y k (ν )=0 ∀1≤i ≤M−1(44)
The above constraints may be recast as the following matrix identity
[H 1 (ν ) … H M (ν )
H 1 (ν+νs ) … H M (ν+νs )⋮ ⋮ ⋮
H 1(ν+(M−1M )ν s) … H M (ν+(M−1
M )νs)]⏟
H ( ν )
[Y 1 ( ν )Y 2 ( ν )⋮
Y M ( ν )]⏟Y ( ν )
=[M νs−1
0⋮0 ] (45)
Please note that the above constraints are strictly valid forν∈ (−0.5νs ,0.5ν s ), as Y k (ν ) vanishes outside this interval. This requirement may be incorporated into (45), to yield the following revised constraints
[H 1 (ν ) … HM (ν )
H1 (ν+νs ) … H M (ν+νs )⋮ ⋮ ⋮
H1(ν+(M−1M )ν s) … H M (ν+(M−1
M )νs)]⏟
H ( ν )
[Y 1 ( ν )Y 2 ( ν )⋮
Y M ( ν )]⏟Y ( ν )
=[M νs−1 rect (νs−1 ν )
0⋮0
] (46)
It is obvious from (45) that the transfer function of the post-filters namely {Y k ( ν ) }k=1M
,
are specified by the first column ofH−1(ν) scaled byM /ν s.
Please note that (45) is identical to Eq-11 in the paper “Multichannel sampling of low-pass
signals” by when k=1, and under the following substitutionsν→ω, H (ν )→A (ω ), Y (ν )→P(ω), T 1=M /νs
Also, the notion that the transfer function of the post- filters are specified by the first column ofH−1(ν) is consistent with Eq-12 of the paper.
(45) is identical to Eqs-(41,42) listed in the book chapter “Superresolution Imaging - Revisited” by Markus E. Testorf and Michael A. Fiddy, under the following substitutions
k→m'−1, i→k ' , Δ νM=νs /M (45) is identical to Eq-(4-21) listed in the book “Systems and Transforms
with Applications in Optics” by A. Papoulis, under the following substitutionsT 0→T ,M→N , 2πν=ω, 2π νs /M=ω0/N
Y k (ν )→Φk (ω) and Φ (ω )≝ rect( ωω0 ) whereω0=2πT 0
PLEASE VERIFY THE LAST ENTRY FORMALLY!!!
Intuitive meaning of (46)According to the GST, perfect reconstruction of f (t ) demands that (46) be satisfied, requiring that each column of the matrix H (ν) is different from a column of zeros. This is only possible when the aggregate bandwidth of the filters H (ν) spans the interval(−0.5 νs ,0.5νs ). In other words, every frequency within the interval
(−0.5 νs ,0.5νs ) must be admitted by one of more of the filters {hk (t ) , yk (t ) }k=1M
.
TODO: Tie above argument with discussion on well-posed GST in “On well-posedness of the Papoulis Generalized Sampling Expansion”…specifically Theorem-1.
Connection between GST and Digital Super Resolution
Figure 5 Interpreting Digital Super Resolution as a variant of the standard GST configuration
Δlo Size of individual pixel in detector
Δhi=Δlo /M Desired pixel size
hopt ( x ) Point Spread Function
H opt (ν)≝F {hopt ( x ) } Optical Transfer Function
Note: The OTF is a band-limited transfer function on account of diffraction
H opt (ν )=0 , for |ν|≥0.5 /Δhi
hdet (x )=rect ( xΔlo )
Detector spatial response (square pixels with 100% fill factor)
H det ( ν )≝F {hdet ( x ) }=Δlo sinc ( Δlo ν ) Detector Transfer Function( accommodate blurring induced by gathering light over the finite area of a single pixel )
obd (x )=o ( x )⊛hopt ( x )⋆hdet(x ) Input to the bank of pre-filters
H k (ν )=exp (− j 2π (k−1)ν Δhi ) Transfer function of k thpre-filter ( due to sub-pixel shifting )
Y k (ν )={exp ( j2π (k−1)ν Δhi )
rect( Δhi2ν) } Transfer function of k thpost-filter
( Ideal low pass filter + linear phase )
The following choice of pre-filters and post-filters guarantee perfect recovery of the imageobd(x), when using a detector with a pixel pitch of Δlo
hk ( x )=δ (x−(k−1)Δhi)⇒H k ( ν )=exp (− j2 π (k−1 ) ν Δhi )
yk ( x )=sinc( x+(k−1)Δhi
Δhi )⇒Yk
(ν )= 1Δhi
exp ( j 2π (k−1 ) ν Δhi ) rect (Δhiν )(47)
The transfer function of the pre-filters correspond to an ideal delay line, while the transfer function of the post-filters correspond to delayed ideal low-pass filters.
It can be readily verified that forν∈( −12 Δhi
, 12 Δhi )
∑k=1
M
H k ( ν )Y k ( ν )= MΔhi
(48)
Likewise, ∀ 1≤ i≤ M−1
∑k=1
M
H k (ν+ iΔlo )Y k (ν )= 1
Δhi∑k=1
M
exp(− j2 π i (k−1 )Δhi
Δlo )= 1Δhi
∑k=1
M
exp (− j 2π i(k−1 )M )= 1
Δhi∑k'=0
M−1
exp(− j2 π i k'
M )= 1Δhi {1−exp (− j2 π iM
M )1−exp(− j2π i
M ) }=0(49)
Note: The shifts in the definition of the pre/post-filters need not be integer multiples of Δhi. They could be arbitrary shifts, so long as the pre/post-filters are perfectly matched.
Expression for the subsampled image in the k thchannelgk ( x )=[obd (x )⊛hk ( x ) ] [∑n∈Z δ (x−nM Δhi) ]=[obd ( x )⊛δ (x−(k−1)Δhi ) ][∑n∈Z δ (x−nM Δhi) ]=[obd ( x−(k−1)Δhi ) ][∑n∈Z δ (x−nM Δhi )]=∑
Details It should be evident from Figure 5 that one can only reconstruct the band-
limited image obd(x) in the absence of noise. This implies that any frequencies lost to optical blurring may never be recovered. Likewise, nulls in H det ( ν ) result in an irreversible loss of the null frequencies. This suggests that clear pixels with large footprint may be not be well suited for perfect recovery of the optical image.
So long as H det ( ν ) is devoid of nulls in the interval ν∈( −12 Δhi
, 12 Δhi ), one can
recover ob(x) using an inverse filter. If one wishes to super-resolve by2×, one could choose
hdet (x )=tri ( x0.5 Δlo )={1−|x|,∨¿ x∨¿0.5 Δlo
0 ,∨x∨≥0.5 ΔloThe resulting transfer function does not have any nulls in the interval
ν∈( −12 Δhi
, 12 Δhi ).
It is worth examining if this idea can be generalized further.
Digital Super Resolution - Reduction to practiceIn practice,
The sampled optical PSF h [i , j ] has finite spatial extent, sayM×N pixels. Likewise, the spatial response of a single detector pixel d [ i , j ] spans G×G higher-resolution pixels. This implies that the cross-correlation Γdh [ i , j ] has a spatial extent of (M+G−1 )× (N+G−1 ) pixels. In conclusion, the intensity of the (m ,n )th detector pixel may be inferred from no more than (M+G−1 )× (N+G−1 ) independently weighted observations ofo (k , l).
The detector has finite area.
TODO Give the finite spatial support of the optical PSF & the detector integration
mask Rewrite the expression for ilo [m ,n ] using matrix algebra, Say we are interested in recovering the image up-to optical blur so that the
following expression is true
~o (x , y )=o ( x , y )⊛hopt ( x , y )⊛ [rect(−xΔhi )rect (− y
Δhi )]This implies that o [k , l ]=~o (k Δhi , l Δhi ) andΓdh [k ,l ]=d [k ,l ]. h [k , l ]=δ [k ,l ], since the optical PSF has been absorbed into the definition of ~o (x , y ). In such a case, one finds that
ilo [m ,n ]={o [k , l ]⊛ d [k , l ] }↓GLikewise,
ilo [m ,n; p ,q ]={o [k−p ,l−q ]⊛d [k , l ] }↓GThe above expression suggests that the intensity of a single low-res pixel is a simple linear combination of G2(G×G) high-res pixel intensities.
Other observationsWe know from (18) that ilo [m ,n ]={o [k ,l ]⊛Γ dh [k , l ] }↓G. In the special case thathopt ( x , y )⋆hdet ( x , y )=δ (x , y), it follows that
ilo [m ,n ]=~o (mG Δhi , nG Δhi) (50)
It is obvious from (50) that the intensity of the (m ,n )th detector pixel is identical to the intensity of the (Gm,Gn)th sample of the band-limited image~o (x , y ).
Lingering Questions How does space-variance in the pixel transfer function of each detector pixel
affect the expression for ilo [m ,n ] ? What happens when the sub-pixel shifts are not integer multiples of Δhi ? What if I want to super-resolve by a factor ¿ Δhi÷ Δlo ? Does our analysis
support this? What if I wanted to super-resolve by a rational number?
Appendix – Useful identities
1. For real signals
[ s1(x , y )⊛s2( x , y) ]⋆ [s3 ( x , y )⊛ s4 ( x , y ) ]=[ s1 (x , y )⊛ s2 ( x , y ) ]⊛ [s3 (−x ,− y )⊛ s4 (−x ,− y ) ]=s1 (x , y )⊛ s2 ( x , y )⊛s3 (−x ,− y )⊛s4 (−x ,− y )=[s1 ( x , y )⊛ s3 (−x ,− y ) ]⊛ [ s2 ( x , y )⊛s4 (−x ,− y ) ]=[s1 ( x , y )⋆ s3 ( x , y ) ]⊛ [ s2 ( x , y )⋆ s4 ( x , y ) ](51)
2. For integer arguments
sinc (i , j )=( sin (πi )πi )( sin (πj )
πj )={1 ,∧i= j0 ,∧i ≠ j
(52)
The above property of the sinc function makes it a cardinal function, which are functions that take the value 0 on the non-zero integers, and take the value 1 at 0.
3. Cross-correlation identities for real signals
f ( x , y )⋆ g(x , y )=∬ f (u , v )g (u+ x , y+v )dudv (53)
δ (x , y )⋆ g ( x , y )=∬δ (u , v ) g (u+x , y+v )dudv=g (x , y ) (54)
δ ( x−x0, y− y 0 )⋆ g ( x , y )=∬δ (u−x0, v− y0 )g (u+ x , y+v )dudv=g (x+x0 , y+ y0) (55)
Accommodating fill factors ¿100 %Replace hdet(x , y ) withhdet−lo ( x , y ).
Δlo pixel size in detectorΔhi desired pixel size
hdet−hi ( x , y )= rect ( xΔhi
, yΔhi ) Response of the (0,0 )th pixel in the higher-
resolution detector.
hdet−lo ( x , y )=rect ( xΔlo )rect( y
Δlo )=rect( xG Δhi )rect ( y
GΔhi )=hdet−hi( xG , yG )Response of the (0,0 )th detector pixel.
hdet−lo (x−mΔlo , y−n Δlo )=rect ( x−mΔlo
Δlo)rect ( y−n Δlo
Δlo)=rect ( x−GmΔhi
G Δhi)rect ( y−Gn Δhi
G Δhi)=hdet−hi( x−GmΔhi
G,y−Gn Δhi
G )Response of the (m ,n )th detector pixel.
Update definition of d [k , l ]
d [k ,l ]≝∬hdet−lo ( x , y )hdet−hi (x−k Δhi , y−l Δhi )dxdy (56)
