Image Enhancement by Regularization Methods Andrey S. Krylov, Andrey V. Nasonov, Alexey S. Lukin...

transcript

Image Enhancement by Image Enhancement by Regularization MethodsRegularization Methods

Andrey S. Krylov, Andrey V. Nasonov, Alexey S. Lukin

Moscow State UniversityFaculty of Computational

Mathematics and CyberneticsLaboratory of Mathematical Methods

of Image Processing

IntroductionIntroduction

Many image processing problems are posed as ill-Many image processing problems are posed as ill-posed inverse problems. To solve these problems posed inverse problems. To solve these problems numerically one must introduce some additional numerically one must introduce some additional information about the solution, such as an assumption information about the solution, such as an assumption on the smoothness or a bound on the norm.on the smoothness or a bound on the norm.

This process was theoretically proven by Russian mathematician

Andrey N. Tikhonov and it is known as regularization.

OutlineOutline

Regularization methodsRegularization methods ApplicationsApplications

– Resampling (interpolation)Resampling (interpolation)– Deringing (Gibbs effect reduction)Deringing (Gibbs effect reduction)– Super-resolutionSuper-resolution

Ill-posed ProblemsIll-posed Problems

Formally, a problem of mathematical physics is called well-posed or well-posed in the sense of Hadamard if it fulfills the following conditions:

1. For all admissible data, a solution exists.

2. For all admissible data, the solution is unique.

3. The solution depends continuously on data.

Ill-posed ProblemsIll-posed Problems

Many problems can be posed as Many problems can be posed as problems of solution of an equationproblems of solution of an equation

AA is a linear continuous operator, is a linear continuous operator, ZZ and and UU are Hilbert spaces are Hilbert spaces

The problem is ill-posed and the The problem is ill-posed and the corresponding matrix for operator corresponding matrix for operator АА in in discrete form is ill-conditioneddiscrete form is ill-conditioned

Point Spread Function (PSF)Point Spread Function (PSF)

Assume:Point light source

PSFPSF ==

Convolution ModelConvolution Model

NotationsNotations

– LL: original image: original image

– KK: the blur kernel (PSF): the blur kernel (PSF)

– NN: sensor noise (white): sensor noise (white)

– BB: input blurred image: input blurred image

Deblur using Convolution Deblur using Convolution Theorem Theorem

B KL KB L

Convolution Theorem:

f g f g

1 /BL K

88/38/38

Deblur using Convolution Deblur using Convolution Theorem Theorem

PSFPSF

BlurredBlurredImageImage

RecoveredRecovered

99/38/38

Noisy caseNoisy case

1 /BL K

DeconvolutionDeconvolutionis unstableis unstable

1 / /K KL B N

1010/38/38

Variational regularization Variational regularization methodsmethods

Tikhonov methodTikhonov methodss

The Residual method (Philips)The Residual method (Philips)

The Quasi-solution method (Ivanov)The Quasi-solution method (Ivanov)

1111/38/38

Variational regularization Variational regularization methodsmethods

Regularization method is determined Regularization method is determined by:by:A) Choice of solution space and of stabilizerA) Choice of solution space and of stabilizer

B) Choice of B) Choice of

C) Method of minimizationC) Method of minimization

A and B determine additional information on A and B determine additional information on problem solution we want to use for solution problem solution we want to use for solution of ill-posed problem to achieve stabilityof ill-posed problem to achieve stability

1212/38/38

OutlineOutline

Regularization methodsRegularization methods ApplicationsApplications

Resampling:Resampling:IntroductionIntroduction

Interpolation is also referred to as Interpolation is also referred to as resampling, resizing or scaling of digital resampling, resizing or scaling of digital imagesimages

Methods:Methods:► Linear non-adaptive Linear non-adaptive (bilinear, bicubic, Lanczos (bilinear, bicubic, Lanczos

interpolation)interpolation)

► Non-linear edge-adaptive Non-linear edge-adaptive (triangulation, gradient (triangulation, gradient methods, NEDI)methods, NEDI)

Regularization method is used to construct a Regularization method is used to construct a non-linear edge-adaptive algorithmnon-linear edge-adaptive algorithm

Resampling:Resampling:Linear and non-linear Linear and non-linear methodmethod

bilinear interpolationbilinear interpolation non-linear methodnon-linear method

Resampling:Resampling:Inverse problemInverse problem

Consider the problem of resampling asConsider the problem of resampling as

Problem: operator A is not invertibleProblem: operator A is not invertible

z is unknown high-resolution image,u is known low-resolution image,A is the downsampling operator which consists offiltering H and decimation D

Resampling:Resampling:RegularizationRegularization

We use Tikhonov-based regularization We use Tikhonov-based regularization methodmethod

wherewhere

Choices of regularizing term (stabilizer)Choices of regularizing term (stabilizer)

– Total VariationTotal Variation

– Bilateral TVBilateral TV

and are shift operators along and are shift operators along xx and and yy axes by axes by ss and and tt pixels pixels respectively, respectively, pp = 1, = 1, γγ = 0.8 = 0.8

Minimization problemMinimization problem

Subgradient methodSubgradient method

Resampling:Resampling:ResultsResults

Linear methodLinear method Regularization-Regularization-basedbased

methodmethod Gibbs Gibbs

phenomenonphenomenon

Introduction to regularizationIntroduction to regularization ApplicationsApplications

Total Variation Approach for Total Variation Approach for DeringingDeringing Gibbs effect is related to Total VariationGibbs effect is related to Total Variation

High TV,very notable Gibbs effect

(ringing)

Low TV

Total Variation Total Variation Regularization methodsRegularization methods

TikhonovTikhonov’s’s approach approach

Rudin, Osher, Fatemi methodRudin, Osher, Fatemi method

Ivanov’s quasi-solution methodIvanov’s quasi-solution method

Global and Local DeringingGlobal and Local Deringing

Two approaches for DeringingTwo approaches for Deringing– Global deringingGlobal deringing

Minimizes TV for entire imageMinimizes TV for entire image In this case, we use In this case, we use Tikhonov regularization Tikhonov regularization

methodmethod No ways to estimate regularization No ways to estimate regularization

parameter, parameter, details outside edges may be lost details outside edges may be lost

– Local deringingLocal deringing Used if we have information on TV for small Used if we have information on TV for small

rectangular areasrectangular areas In this case, we use In this case, we use Ivanov’s quasi-solution Ivanov’s quasi-solution

method method for small overlapping blocksfor small overlapping blocks

Deringing after Deringing after interpolationinterpolation

Deringing after interpolationDeringing after interpolation– We know information on TV for blocks of initial We know information on TV for blocks of initial

image to be resampledimage to be resampled– We suggest that TV does not change after We suggest that TV does not change after

image interpolationimage interpolation

Thus we have real algorithm to find regularization parameter for deringing

after image resampling task

MinimizationMinimization

Tikhonov regularization methodTikhonov regularization method– Subgradient methodSubgradient method

Quasi-solution methodQuasi-solution method– 1D Conditional gradient method 1D Conditional gradient method (there is no (there is no

effective 2D implementation)effective 2D implementation)– In 2D case, we divide an image into a set of In 2D case, we divide an image into a set of

rows and process these rows by 1D method, rows and process these rows by 1D method, next we do the same with columns and finally next we do the same with columns and finally we average these resultswe average these results

MinimizationMinimization

Conditional gradient methodConditional gradient method– Conditional gradient method is used to Conditional gradient method is used to

minimize a convex functional on a convex minimize a convex functional on a convex compact set. The key idea of this method is that compact set. The key idea of this method is that step directions are chosen among the vertices step directions are chosen among the vertices of the set of constraints, so we do not fall of the set of constraints, so we do not fall outside this set during minimization processoutside this set during minimization process

– Conditional gradient method is effective only for Conditional gradient method is effective only for small imagessmall images, so it is used for local deringing , so it is used for local deringing onlyonly

Resampling + Deringing:Resampling + Deringing:PSNR ResultsPSNR Results

5.0 0 5.0 0

After resampling 28.38Global

deringingLocal

deringingConditional gradient method 23.33 29.24Subgradient method 28.71 28.64

A set of 100 nature and architecture images with 400x300 resolution (11x11 blocks, 1813 per image) was used to test the methods.

We downsampled the images by 2x2 using Gauss blur with radius 0.7 and then upsampled them by our regularization algorithm. Next we applied deringing methods and compared the results with initial images.

Resampling + Deringing:Resampling + Deringing:ResultsResults

regularization-basedregularization-basedinterpolationinterpolation

application of quasi-application of quasi-solution methodsolution method

Resampling + Deringing:Resampling + Deringing:ResultsResults

Source image, upsampled by box filterLinear interpolationRegularization-based methodRegularization-based interpolation +quasi-solution deringing method

Introduction to regularizationIntroduction to regularization ApplicationsApplications

Super-Resolution:Super-Resolution:IntroductionIntroduction

The problem of super-resolution is to recover a The problem of super-resolution is to recover a high-resolution image from a set of several high-resolution image from a set of several degraded low-resolution images degraded low-resolution images

Super-resolution methodsSuper-resolution methods– Learning-basedLearning-based – single image super-resolution, – single image super-resolution,

learning database (matching between low- and high-learning database (matching between low- and high-resolution images) resolution images)

– Reconstruction-basedReconstruction-based – use only a set of low-resolution – use only a set of low-resolution images to construct high-resolution image images to construct high-resolution image

Super-Resolution:Super-Resolution:Inverse ProblemInverse Problem

The problem of super-resolution is posed as The problem of super-resolution is posed as error minimization problemerror minimization problem

z z – reconstructed high-resolution image– reconstructed high-resolution image vvkk – – kk-th low-resolution input image-th low-resolution input image

AAkk – downsampling operator, it includes – downsampling operator, it includes motion informationmotion information

Super-Resolution:Super-Resolution:Downsampling operatorDownsampling operator

AAkk – downsampling operator – downsampling operator

HHcamcam – camera lens blur (modeled by Gauss – camera lens blur (modeled by Gauss filter)filter)

HHatmatm – atmosphere turbulence effect (neglected) – atmosphere turbulence effect (neglected) nn – noise (ignored) – noise (ignored) FFkk – warping operator – motion deformation – warping operator – motion deformation

Super-Resolution:Super-Resolution:Warping operatorWarping operator

Warping operator Warping operator FFkk

Super-Resolution:Super-Resolution:RegularizationRegularization

The problem is ill-posed, and we use The problem is ill-posed, and we use Tikhonov regularization approach (same as in Tikhonov regularization approach (same as in resampling)resampling)

where ,where ,

Minimization by subgradient methodMinimization by subgradient method

Super-Resolution:Super-Resolution:ResultsResults

Linear methodPixel replication

Non-linear method

Super-resolutionFace super-resolution for the

factor of 4 and 10 input images

Source images

Super-Resolution:Super-Resolution:ResultsResults

The reconstruction of an image from a sequenceThe reconstruction of an image from a sequence

examples of input frames (of total 14)

linearly interpolated single frame

super-resolution result

Super-Resolution for VideoSuper-Resolution for Video

Super-Resolution

For every frame, we take current frame, 3 previous and 3 next frames. Then we process it by

super-resolution.

current frame

Super-Resolution:Super-Resolution:Results for VideoResults for Video

Nearest neighbor interpolation

Super-Resolution

Super-Resolution for video for a factor of 4

Bilinear interpolation Super-Resolution

Bicubic interpolation Super-Resolution

ConclusionConclusion

Increasing CPU and GPU power Increasing CPU and GPU power makes regularization methods makes regularization methods more and more important in more and more important in image processingimage processing

Regularization is a very powerful Regularization is a very powerful tool but each specific image tool but each specific image processing problem needs its own processing problem needs its own regularization methodregularization method

Thank you!Thank you!

http://imaging.cs.msu.ru/http://imaging.cs.msu.ru/

Image Enhancement by Regularization Methods Andrey S. Krylov, Andrey V. Nasonov, Alexey S. Lukin...

Documents