Some Blind Deconvolution Techniques in Image Processing

Tony Chan

Math Dept., UCLA

Astronomical Data Analysis Software & SystemsConference Series 2004

Pasadena, CA, October 24-27, 2004

Joint work with Frederick Park and

Andy M. Yip

2

OutlinePart I:

Total Variation Blind Deconvolution

Part II:Simultaneous TV Image Inpainting and

Blind Deconvolution

Part III:Automatic Parameter Selection for TV

Blind Deconvolution

Caution: Our work not developed specifically for Astronomical images

3

Blind Deconvolution Problem

= +

Observed image

Unknown true image

Unknown point spread function

Unknown noise

Goal: Given uobs, recover both uorig and k

obsu origu k

4

Typical PSFsPSFs w/ sharp edges:

PSFs w/ smooth transitions

5

Total Variation Regularization

dxxuuTV )()(

• Deconvolution ill-posed: need regularization

• Total variation Regularization:

dxxkkTV )()(

• The characteristic function of D with height h (jump):

• TV = Length(∂D)h• TV doesn’t penalize jumps• Co-area Formula:

D

h drdsfdxufnR

ru

)(||}{

6

TV Blind Deconvolution Model

TVTVobsku

kuukukuF 21

2

,),(min

),(),(,1),(,0, yxkyxkdxdyyxkku Subject to:

Objective:

(C. and Wong (IEEE TIP, 1998))

1 determined by signal-to-noise ratio

2 parameterizes a family of solutions, corresponds to different spread of the reconstructed PSF

• Alternating Minimization Algorithm:

• Globally convergent with H1 regularization.

),(min),(

),(min ),(

111

1

kuFkuF

kuFkuF

n

k

nn

n

u

nn

7

Blind v.s. non-Blind Deconvolution

Observed Image noise-free

• An out-of-focus blur is recovered automatically

• Recovered blind deconvolution images almost as good as non-blind

• Edges well-recovered in image and PSF

non-Blind

Recovered Image PSF

Blind

1 = 2106, 2 = 1.5105

Clean image

True PSF

8

Blind v.s. non-Blind Deconvolution: High Noise

Observed Image SNR=5 dB

non-Blind

Clean image

True PSF

Blind

• An out-of-focus blur is recovered automatically

• Even in the presence of high noise level, recovered images from blind deconvolution are almost as good as those recovered with the exact PSF

1 = 2105, 2 = 1.5105

9

Controlling Focal-Length Recovered Images are 1-parameter family w.r.t. 2

Recovered Blurring Functions(1 = 2106)

0 1107 1105 11042:

The parameter 2 controls the focal-length

10

Generalizations to Multi-Channel Images

• Inter-Channel Blur Model– Color image (Katsaggelos et al, SPIE 1994):

Bobs

Gobs

Robs

B

G

R

kkk

kkk

kkk

u

u

u

u

u

u

HHH

HHH

HHH

noise

122

212

221

75

71

71

71

75

71

71

71

75

obsk unoiseuH

k1: within channel blur

k2: between channel blur

m-channel TV-norm (Color-TV)

(C. & Blomgren, IEEE TIP ‘98)

2

2

1

2)(

m

iTVim

kkTV

m

iTVim uuTV

1

2)(

11

Original image

Out-of-focus blurred blind non-blind

Gaussian blurred blind non-blind

Examples of Multi-Channel Blind Deconvolution(C. and Wong (SPIE, 1997))

• Blind is as good as non-blind

• Gaussian blur is harder to recover (zero-crossings in frequency domain)

12

TV Blind Deconvolution Patented!

13

Outline

Part I:Total Variation Blind Deconvolution

Part II:Simultaneous TV Image Inpainting and

Blind Deconvolution

Part III:Automatic Parameter Selection for TV

Blind Deconvolution

14

TV Inpainting Model(C. & Shen SIAP 2001)

EDE

dxdyuudxdyuuJ ,||2

||][ 20

,0)(||

0

uuu

ue

.0; ,,DzEz

e

Graffiti Removal

Scratch Removal

15

Images Degraded by Blurring and Missing Regions

• Blur– Calibration errors of devices– Atmospheric turbulence– Motion of objects/camera

• Missing regions– Scratches– Occlusion– Defects in films/sensors

+

16

Problems with Inpaint then Deblur

• Inpaint first reduce plausible solutions

• Should pick the solution using more information

Original Signal Blurring func.

Original Signal Blurring func.

Blurred Signal

Blurred Signal

=

=

Blurred + Occluded

Blurred + Occluded

=

17

Problems with Deblur then Inpaint

• Different BC’s correspond to different image intensities in inpaint region.

• Most local BC’s do not respect global geometric structures

Original Occluded Support of PSF

Dirichlet Neumann Inpainting

18

The Joint Model

• Do --- the region where the image is observed

• Di --- the region to be inpainted

• A natural combination of TV deblur + TV inpaint

• No BC’s needed for inpaint regions

• 2 parameters (can incorporate automatic parameter selection techniques)

Inpainting take place

Coupling of inpainting & deblur

19

Simulation Results (1)Degraded Restored

Zoom-in

• The vertical strip is completed

• Not completed

• Use higher order inpainting methods

• E.g. Euler’s elastica, curvature driven diffusion

20

Simulation Results (2)

Observed Restored

Deblur then inpaint

(many artifacts)

Inpaint then deblur

(many ringings)

Original

21

Boundary Conditions for Regular Deblurring

Original image domain and

artificial boundary outside the scene

Dirichlet B.C.

Periodic B.C.

Neumann B.C.

Inpainting B.C.

22

23

Outline

Part I:Total Variation Blind Deconvolution

Part II:Simultaneous TV Image Inpainting and

Blind Deconvolution

Part III:Automatic Parameter Selection for TV

Blind Deconvolution (Ongoing Research)

24

Automatic Blind Deblurring (ongoing research)

• Recovered images: 1-parameter family wrt 2

• Consider external info like sharpness to choose optimal 2

Problem: Find 2 automatically to recover best u & k

SNR = 15 dB20 40 60 80 100 120

20

40

60

80

100

120

20 40 60 80 100 120

20

40

60

80

100

120

Clean image

observed image

25

Motivation for Sharpness & Support

• Sharpest image has large gradients• Preference for gradients with small support

20 40 60 80 100 120

20

40

60

80

100

120

20 40 60 80 100 120

20

40

60

80

100

120

20 40 60 80 100 120

20

40

60

80

100

120

20 40 60 80 100 120

20

40

60

80

100

120

20 40 60 80 100 120

20

40

60

80

100

120

20 40 60 80 100 120

20

40

60

80

100

120

20 40 60 80 100 120

20

40

60

80

100

120

20 40 60 80 100 120

20

40

60

80

100

120

u

|| u

|| u

20 40 60 80 100 120

20

40

60

80

100

120

20 40 60 80 100 120

20

40

60

80

100

120

20 40 60 80 100 120

20

40

60

80

100

120

20 40 60 80 100 120

20

40

60

80

100

120

Support of

26

Proposed Sharpness Evaluator

• F(u) small => sharp image with small support• F(u)=0 for piecewise constant images• F(u) penalizes smeared edges

||ofsupport of Area)( uuF

20 40 60 80 100 120

20

40

60

80

100

120

20 40 60 80 100 120

20

40

60

80

100

120

20 40 60 80 100 120

20

40

60

80

100

120

20 40 60 80 100 120

20

40

60

80

100

120

u

|| u20 40 60 80 100 120

20

40

60

80

100

120

20 40 60 80 100 120

20

40

60

80

100

120

20 40 60 80 100 120

20

40

60

80

100

120

20 40 60 80 100 120

20

40

60

80

100

120

Support of

27

Planets Example

20 40 60 80 100 120

20

40

60

80

100

120

20 40 60 80 100 120

20

40

60

80

100

120

Rel. errors in u (blue) and k (red) v.s. 2

Proposed Objective v.s. 2

Optimal Restored Image Auto-focused Image

The minimum of the sharpness function agrees with that of the

rel. errors of u and k

(minimizer of sharpness func.)

(minimizer of rel. error in u)

1=0.02 (optimal)

105

106

107

0

0.5

1

1.5

105

106

107

1500

2000

2500

3000

3500

4000

4500

5000

28

Satellite ExampleRel. errors in u (blue) and

k (red) v.s. 2

Proposed Objective v.s. 2

Optimal Restored Image Auto-focused Image

The minimum of the sharpness function agrees with that of the

rel. errors of u and k

(minimizer of sharpness func.)

(minimizer of rel. error in u)

1=0.3 (optimal)

105

106

107

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

20 40 60 80 100 120

20

40

60

80

100

120

105

106

107

2000

3000

4000

5000

6000

7000

8000

9000

10000

11000

20 40 60 80 100 120

20

40

60

80

100

120

29

Potential Applications to Astronomical Imaging

• TV Blind Deconvolution– TV/Sharp edges useful?– Auto-focus: appropriate objective function?– How to incorporate a priori domain knowledge?

• TV Blind Deconvolution + Inpainting– Other noise models: e.g. salt-and-pepper noise

30

References

1. C. and C. K. Wong, Total Variation Blind Deconvolution, IEEE Transactions on Image Processing, 7(3):370-375, 1998.

2. C. and C. K. Wong, Multichannel Image Deconvolution by Total Variation Regularization, Proc. to the SPIE Symposium on Advanced Signal Processing: Algorithms, Architectures, and Implementations, vol. 3162, San Diego, CA, July 1997, Ed.: F. Luk.

3. C. and C. K. Wong, Convergence of the Alternating Minimization Algorithm for Blind Deconvolution, UCLA Mathematics Department CAM Report 99-19.

4. R. H. Chan, C. and C. K. Wong, Cosine Transform Based Preconditioners for Total Variation Deblurring, IEEE Trans. Image Proc., 8 (1999), pp. 1472-1478

5. C., A. Yip and F. Park, Simultaneous Total Variation Image Inpainting and Blind Deconvolution, UCLA Mathematics Department CAM Report 04-45.