Computational 3D Imaging:Sparse Recovery and

PSF Engineering

Bob Plemmons

Wake Forest University (Emeritus)

Imaging, Vision and Learning based on Optimization and PDEs - Bergen Norway 2016

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 1

Outline

Two Related Projects1. Compressed sensing 3D data sparse

reconstruction with nonconvex regularizationI 3D information from 2D images

2. Single frame (snapshot) 3D imaging based onPoint Spread Function (PSF) engineering

I Depth from defocusI Wavefront phase coding

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 2

Some Goals of 3D Optical Imaging

I To seek the complete structure of objects in oursurroundings

I To extract information from the image data such as:

I Object distance from the camera as part of 3Dlocalization

I Brightness, orientation, and shape of objects

I Includes our work on specular and diffuse BRDFcomputations for object characterization usinghyperspectral and polarization imaging

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 3

Hyperpectral Imaging (HSI)

• Spectral imagers capture a 3D datacube (tensor)containing:

I 2D spatial information: x-yI 1D spectral information at each spatial

location: λ

• Pixel intensity varies with wavelength bands -provides a spectral trace of intensity values.

•We add polarization to identify object shape,orientation and metallic surfaces.

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 4

HSI: Spectral (beyond RGB) Imaging atWavelengths λ

Figure 1: λ generally ranges between 400 and 2500 nanometers.

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 5

Using the Spectra λ

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 6

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 7

Some Applications of Hyperspectral Imaging (HSI)and Polarization

I Environmental remote sensing, e.g.,monitoring disasters, chemical/oil spills, etc.

I Military target discriminationI AstrophysicsI Biomedical optics, medical microscopy, etc.I Remote surveillance for defense & security,

e.g., imaging a compound in western Pakistan

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 8

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 9

Space Situational Awareness, Military and Commercial

Figure 2: Application: Satellites & Large Debris Objects AroundEarth. (About 1,900 active satellites to maintain, and debris tomonitor - international cooperation, huge expenditures)

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 10

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 11

Current AFOSR Project

• “Statistical Image Analysis and Applications to High DimensionalImaging for Improved SSA,” Overall PI S. Prasad, Physicist UNM ,2015 - 2018.

• Develop algorithms and statistical performance metrics asdesign tools for high dimensional object tracking systems.Tracking debris: 3D imaging with PSF engineering.

A US Air Force Observatory on Haleakala in Maui

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 12

• Spectro-Polarimetric Imaging (SPI)

I 3D hyperspectral & 1D polarization. Data is a 4D tensor.I Spectral traces identify materials.I Polarizations identify object shape, orientation, glint.I Object characterization using spectrally compressive

polarimetric image data, funded by AFOSR grant to UNM(Physics), Duke (ECE) and WFU (Math & CS). Snapshotspectro-polarimetric cameras developed in ECE at Duke.

Sparse recovery by “nonconvex optimization methods.”

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 13

Coded Aperture Snapshot Spectral Compressive Sensor

• Dave Brady et al., Duke U., part of AFOSR SOI project.

g(x , y) =∫λ

Cλ(x , y)f (x , y , λ)dλ+ ελ(x , y).

Cλ(x , y) = system function. Compressive HSI sensing - requiresreconstruction. 3D information from 2D images.

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 14

Spectro-Polarimetric Camera: Duke, UNM, WFU AFOSR Proj.

System modulates 4D tensor array images onto a 2D detector(matrix). Reconstruct 4 polarized images.

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 15

Spectro-Polarimetric Compressive Sensing

• Spatial Light Modulator (SLM)-based - snapshotspectro-polarimetric imager forward model.

g(x , y) =∫λ

Cµ(x , y , λ)fµ(x , y , λ)dλ+ ελ,µ(x , y).

Cµ = system function, µ = linear polarization variable, λ =wavelength, solve inverse problem for fµ.

minimize[

12‖Hµfµ − g‖22 + τ‖fµ‖pp

]. (1)

Hµ = system matrix MN × 4MN for each of 4 polarizations, gµ =vectorized 2D measurement matrix, fµ = vectorized 3D recoveryarray. Data compression factor = 4, for each polarization angle.

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 16

Some References

Zhang, Plemmons, Kittle, Brady, Prasad. “Jointsegmentation and reconstruction of hyperspectral datawith compressed measurements”. Applied Optics, 2011.

Tsai, Yuan, Carin, and Brady, “Spatial light modulator basedspectral polarization imaging”, OSA COSI Proc., 2014.

Plemmons, Prasad, Pauca, “Spectro-polarimetric imagingthrough atmospheric turbulence”. OSA SRS Proc., 2014.

L. Adhikari, J. Erway, R. Marcia, R. Plemmons. Trust-regionmethods for nonconvex sparse recovery optimizationProc. International Symposium on Information Theory and ItsApplications (ISITA), IEEE Xplore, 2016.

L. Adhikari, J. Erway, R. Marcia, R. Plemmons. Trust-RegionMethods for Nonconvex Sparse Recovery in 3D Imaging,in progress, 2016.

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 17

Overview of TrustSpa-`p Method for Non-ConvexRegularization in Sparse Data Recovery

Solve the `2-`p, 0 < p < 1, sparse recovery problem:

minimize f ∈ Rn(

12‖Af − y‖22 + τ‖f‖pp

)(2)

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 18

Unconstrained Differentiable Problem

I Reformulated (2) as a smooth unconstrained optimizationproblem using a change of variables f = u− v , where u, v ≥ 0

I We apply a limited-memory trust-region method of solution

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 19

TrustSpa-`p Method for Non-Convex Regularization in SparseData Recovery

I Our numerical results indicate proposed non-convexTrustSpa-`p approach eliminates spurious solutions effectively

I Faster for our test problems, in comparison to TwIST (TVreg.), and GPSR.

I See the reference below for the use of ‖∇f‖pp instead of ‖f‖ppas the regularization term.

M. Hintermuller and T. Wu.“Nonconvex TV q-models in image restoration: Analysisand a trust-region regularizationbased superlinearlyconvergent solver,” SIAM Journal on Imaging Sciences,vol. 6, no. 3, pp. 13851415, 2013.

Al interesting open problem is the use of TV q-models in ourTrustSpa-`p algorithm.

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 20

Prototype camera, data acquisition by Tsung-HanTsai, including polarization

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 21

Recovery Process at Duke (with TwIST)

(We tested TrustSpa-`p, for sparse recovery, similar accuracybut much faster than TwIST and GPSR.)

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 22

Recovery with TrustSpa-`p

minimize f

[12‖Hf − g‖22 + τ‖f‖pp

]. (3)

H = system matrix MN × 4MN , g = vectorized 2D measurementmatrix, f = vectorized 3D recovered array. Much faster thanTwIST. Data compression factor = 4, for each polarization angle.

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 23

Work in Progress: Non-lab Data to Test

I (a) Unpolarized reference in RGB.I (b) Compressed spectro-polarimetric measurement.I (c) - (f) Reference polarized image channels in RGB.I (g) - (j) Reconstructions from (b) by TwIST (TV reg) Alg.

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 24

2nd Topic: 3D Imaging using aRotating Point Spread Function Approach toDepth from Defocus

I Part of current AFOSR project: characterizing and monitoringspace objects, including debris swarms.

I Additional applications in high-resolution microscopy.

I Fresnel Lens-type spiral phase mask - patent by ovearall PI S.Prasad.

I PSF rotation with change of focus, enables target localizationin 3D. Imaging swarms (clouds) of point sources.

I Need near real-time optimization with single snapshot image.

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 25

References and Related Papers

S. Prasad“Rotating Point Spread Function via Pupil-phase Engineering”Optics Letters, 2013.

Z. Yu and S. Prasad“High-numerical-aperture Microscopy with a Rotating PSF”J. Opt. Soc. Am. A , 2016.

P. Pauca, S. Prasad, R. Plemmons, T. Torgersen“Optimization Methods for Point Source Localization from PSFEngineering,” in progress, 2016.

Two recent papers on finding depth from focus or defocus:M. Moeller, M. Benning,C. Schonlieb, D. Cremers“Variational Depth From Focus Reconstruction”IEEE Trans. Image Proc., 2015.

S. Suwajanakorn, C. Hernandez, S. Seitz“Depth from focus with your mobile phone” (first non-lab implementation)Proc of the IEEE Conf. on Computer Vision and Pattern Recognition, 2015.

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 26

Introduction: A Bit of Imaging Physics

I 3D snapshot imaging via Rotating PSF using Orbital AngularMomentum (OAM) states of light beams

I PSF image rotation via “twisted” wavefront phase

I Based on either Gauss-Laguerre beams, or non-diffractingBessel beams (our approach)

I Can create beam rotation with spiral phase retardation in thebeam path

I Use OAM to rotate PSF image with changing axial depth

I 3D information encoded in defocused 2D imagesI Fresnel-zone subdivision of pupil using phase mask achieves

depth encoding

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 27

Beam Spiraling via Pupil Phase - Leads to Rotating PSF

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 28

Characterizing the Rotating PSF

I Defocus blurring is a measure of how the PSF changes forobjects at different distances from the lens

I Physically, defocus parameter is phase error, compared to acorrectly focusing wave, at edge of aperture (in radians)

I Amount of defocus quantified by defocus parameter

where λ = wavelength of light, R = radius of the exit pupil,zobj = infocus object distance, z ′

obj = actual object distance.I The PSF for defocused objects can be found using the

generalized pupil function

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 29

Point Source Images with Spiral Phase Mask

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 30

3D Point Source Images with Conventional Camera andRotating PSF

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 31

PSF Engineering in MicroscopyI Imaging and localizing single molecules with high accuracy in

a 3D volume is a challenging task.I Using the rotating PSF can provide an effective localization

strategy and achieve an increased depth for single moleculeimaging.

I Snapshot 3D localization can be used to see how structuresfold and deform. C. Cremers group PSF eng., Heidelberg

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 32

Setup for Microscopy

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 33

Danger to Space Assets - Clouds of Debris asPoint Sources

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 34

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 35

Setup for Space Object Monitoring

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 36

Image Recovery - 3D Point Source LocalizationInitial cost function:

C(r , z, f ) =12

[‖

P∑1

fiH(ri , zi)−G‖22 + τ‖(r , z, f )‖qq

](4)

Here:

I P = number of point sources,I r = (r1, . . . , rP), with ri = (xi , yi), = transverse location vector,I z = (z1, . . . , zP) = depth vector,I f = (f1, . . . , fp) = point source energy fluxI G = 2D camera image data matrix,I H(ri : zi) = estimated rotating PSF (blur) matrix for the ith

point source.

Vectorize the images for computation. Choosing 0 < q < 1 andalso experimenting with TV reg.

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 37

CommentsI Cost function C is nonconvex with multiple local minimizers,

but has global minimizer, based on gradient computations.I Requires initial estimates (xi , yi , zi , fi) to be chosen close to

the global minimizer.I Current work on replacing C by convex surrogate function in

progress. Some success by adding a center of massmatching constraint on G and each H(ri : zi) term.

I Resulting surface plot of transformed cost function.

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 38

Preliminary Computations: Computed image PSFs withpoint sources at various depths:

∑121 fiH(ri, zi)

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 39

Preliminary Computations: Point source localization results,P = 4True 3D locations - green, Computed 3D locations - blue

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 40

Preliminary Computations: Point source flux results, P = 4True - green, Computed - blue (relatively accurate)

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 41

Another Project

U.S. National Geospatial-Intelligence Agency (NGA). InvolvesPauca, Ple., Torgersen (WFU) and Prasad (UNM).

“Bayesian Inference on Convolutional Neural Networks forObject Characterization and Classification in Multimodal andCompressed Sensing Data”

Pixel-level fusion of HSI and LiDAR data

Targets placed in scene by sponsor for test purposes.

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 42

Another ProjectI Hong Kong Research Grants Council (RGC). PI Raymond

Chan (CUHK). Also involves Ple. (WFU) and Wenxing Zhang(UESTC - Chengdu)“Mathematics in the Estimation of Point-spreadFunctions in Ground-based Astronomy throughTurbulence”

(a) φ = wavefront phase, px and py gra-dients in x and y directions

(b) Reconstructions

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 43

Summary

Related 3D Imaging Projects

1. Overview of TrustSpa-`p Method for Non-ConvexRegularization in Sparse Data Recovery.

• Applications to sparse recovery of spectro-polarimetriccompressed sensing 3D images.

2. 3D Imaging using a Rotating Point Spread Function Approachto Depth from Defocus.

• Applications to microscopy and space debris monitoring.

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 44

Thank You!

Bob Plemmons Computational 3D Imaging: Sparse Recovery and PSF Engineering 45