Date post: | 31-Mar-2015 |
Category: |
Documents |
Upload: | mikaela-stear |
View: | 217 times |
Download: | 0 times |
Image Enhancement by Image Enhancement by Regularization MethodsRegularization Methods
Andrey S. Krylov, Andrey V. Nasonov, Alexey S. Lukin
Moscow State UniversityFaculty of Computational
Mathematics and CyberneticsLaboratory of Mathematical Methods
of Image Processing
IntroductionIntroduction
Many image processing problems are posed as ill-Many image processing problems are posed as ill-posed inverse problems. To solve these problems posed inverse problems. To solve these problems numerically one must introduce some additional numerically one must introduce some additional information about the solution, such as an assumption information about the solution, such as an assumption on the smoothness or a bound on the norm.on the smoothness or a bound on the norm.
This process was theoretically proven by Russian mathematician
Andrey N. Tikhonov and it is known as regularization.
OutlineOutline
Regularization methodsRegularization methods ApplicationsApplications
– Resampling (interpolation)Resampling (interpolation)– Deringing (Gibbs effect reduction)Deringing (Gibbs effect reduction)– Super-resolutionSuper-resolution
Ill-posed ProblemsIll-posed Problems
Formally, a problem of mathematical physics is called well-posed or well-posed in the sense of Hadamard if it fulfills the following conditions:
1. For all admissible data, a solution exists.
2. For all admissible data, the solution is unique.
3. The solution depends continuously on data.
Ill-posed ProblemsIll-posed Problems
Many problems can be posed as Many problems can be posed as problems of solution of an equationproblems of solution of an equation
AA is a linear continuous operator, is a linear continuous operator, ZZ and and UU are Hilbert spaces are Hilbert spaces
The problem is ill-posed and the The problem is ill-posed and the corresponding matrix for operator corresponding matrix for operator АА in in discrete form is ill-conditioneddiscrete form is ill-conditioned
Point Spread Function (PSF)Point Spread Function (PSF)
Assume:Point light source
PSFPSF ==
Convolution ModelConvolution Model
NotationsNotations
– LL: original image: original image
– KK: the blur kernel (PSF): the blur kernel (PSF)
– NN: sensor noise (white): sensor noise (white)
– BB: input blurred image: input blurred image
+
Deblur using Convolution Deblur using Convolution Theorem Theorem
B KL KB L
/BL K
Convolution Theorem:
f g f g
1 /BL K
LB K
88/38/38
Deblur using Convolution Deblur using Convolution Theorem Theorem
PSFPSF
BlurredBlurredImageImage
RecoveredRecovered
99/38/38
Noisy caseNoisy case
1 /BL K
DeconvolutionDeconvolutionis unstableis unstable
1 / /K KL B N
1010/38/38
Variational regularization Variational regularization methodsmethods
Tikhonov methodTikhonov methodss
The Residual method (Philips)The Residual method (Philips)
The Quasi-solution method (Ivanov)The Quasi-solution method (Ivanov)
1111/38/38
Variational regularization Variational regularization methodsmethods
Regularization method is determined Regularization method is determined by:by:A) Choice of solution space and of stabilizerA) Choice of solution space and of stabilizer
B) Choice of B) Choice of
C) Method of minimizationC) Method of minimization
A and B determine additional information on A and B determine additional information on problem solution we want to use for solution problem solution we want to use for solution of ill-posed problem to achieve stabilityof ill-posed problem to achieve stability
1212/38/38
OutlineOutline
Regularization methodsRegularization methods ApplicationsApplications
– Resampling (interpolation)Resampling (interpolation)– Deringing (Gibbs effect reduction)Deringing (Gibbs effect reduction)– Super-resolutionSuper-resolution
Resampling:Resampling:IntroductionIntroduction
Interpolation is also referred to as Interpolation is also referred to as resampling, resizing or scaling of digital resampling, resizing or scaling of digital imagesimages
Methods:Methods:► Linear non-adaptive Linear non-adaptive (bilinear, bicubic, Lanczos (bilinear, bicubic, Lanczos
interpolation)interpolation)
► Non-linear edge-adaptive Non-linear edge-adaptive (triangulation, gradient (triangulation, gradient methods, NEDI)methods, NEDI)
Regularization method is used to construct a Regularization method is used to construct a non-linear edge-adaptive algorithmnon-linear edge-adaptive algorithm
Resampling:Resampling:Linear and non-linear Linear and non-linear methodmethod
bilinear interpolationbilinear interpolation non-linear methodnon-linear method
Resampling:Resampling:Inverse problemInverse problem
Consider the problem of resampling asConsider the problem of resampling as
Problem: operator A is not invertibleProblem: operator A is not invertible
z is unknown high-resolution image,u is known low-resolution image,A is the downsampling operator which consists offiltering H and decimation D
Resampling:Resampling:RegularizationRegularization
We use Tikhonov-based regularization We use Tikhonov-based regularization methodmethod
wherewhere
Resampling:Resampling:RegularizationRegularization
Choices of regularizing term (stabilizer)Choices of regularizing term (stabilizer)
– Total VariationTotal Variation
– Bilateral TVBilateral TV
and are shift operators along and are shift operators along xx and and yy axes by axes by ss and and tt pixels pixels respectively, respectively, pp = 1, = 1, γγ = 0.8 = 0.8
Resampling:Resampling:RegularizationRegularization
Minimization problemMinimization problem
Subgradient methodSubgradient method
Resampling:Resampling:ResultsResults
Linear methodLinear method Regularization-Regularization-basedbased
methodmethod Gibbs Gibbs
phenomenonphenomenon
Image Enhancement by Image Enhancement by Regularization MethodsRegularization Methods
Introduction to regularizationIntroduction to regularization ApplicationsApplications
– Resampling (interpolation)Resampling (interpolation)– Deringing (Gibbs effect reduction)Deringing (Gibbs effect reduction)– Super-resolutionSuper-resolution
Total Variation Approach for Total Variation Approach for DeringingDeringing Gibbs effect is related to Total VariationGibbs effect is related to Total Variation
High TV,very notable Gibbs effect
(ringing)
Low TV
Total Variation Total Variation Regularization methodsRegularization methods
TikhonovTikhonov’s’s approach approach
Rudin, Osher, Fatemi methodRudin, Osher, Fatemi method
Ivanov’s quasi-solution methodIvanov’s quasi-solution method
Global and Local DeringingGlobal and Local Deringing
Two approaches for DeringingTwo approaches for Deringing– Global deringingGlobal deringing
Minimizes TV for entire imageMinimizes TV for entire image In this case, we use In this case, we use Tikhonov regularization Tikhonov regularization
methodmethod No ways to estimate regularization No ways to estimate regularization
parameter, parameter, details outside edges may be lost details outside edges may be lost
– Local deringingLocal deringing Used if we have information on TV for small Used if we have information on TV for small
rectangular areasrectangular areas In this case, we use In this case, we use Ivanov’s quasi-solution Ivanov’s quasi-solution
method method for small overlapping blocksfor small overlapping blocks
Deringing after Deringing after interpolationinterpolation
Deringing after interpolationDeringing after interpolation– We know information on TV for blocks of initial We know information on TV for blocks of initial
image to be resampledimage to be resampled– We suggest that TV does not change after We suggest that TV does not change after
image interpolationimage interpolation
Thus we have real algorithm to find regularization parameter for deringing
after image resampling task
MinimizationMinimization
Tikhonov regularization methodTikhonov regularization method– Subgradient methodSubgradient method
Quasi-solution methodQuasi-solution method– 1D Conditional gradient method 1D Conditional gradient method (there is no (there is no
effective 2D implementation)effective 2D implementation)– In 2D case, we divide an image into a set of In 2D case, we divide an image into a set of
rows and process these rows by 1D method, rows and process these rows by 1D method, next we do the same with columns and finally next we do the same with columns and finally we average these resultswe average these results
MinimizationMinimization
Conditional gradient methodConditional gradient method– Conditional gradient method is used to Conditional gradient method is used to
minimize a convex functional on a convex minimize a convex functional on a convex compact set. The key idea of this method is that compact set. The key idea of this method is that step directions are chosen among the vertices step directions are chosen among the vertices of the set of constraints, so we do not fall of the set of constraints, so we do not fall outside this set during minimization processoutside this set during minimization process
– Conditional gradient method is effective only for Conditional gradient method is effective only for small imagessmall images, so it is used for local deringing , so it is used for local deringing onlyonly
Resampling + Deringing:Resampling + Deringing:PSNR ResultsPSNR Results
5.0 0 5.0 0
After resampling 28.38Global
deringingLocal
deringingConditional gradient method 23.33 29.24Subgradient method 28.71 28.64
A set of 100 nature and architecture images with 400x300 resolution (11x11 blocks, 1813 per image) was used to test the methods.
We downsampled the images by 2x2 using Gauss blur with radius 0.7 and then upsampled them by our regularization algorithm. Next we applied deringing methods and compared the results with initial images.
Resampling + Deringing:Resampling + Deringing:ResultsResults
regularization-basedregularization-basedinterpolationinterpolation
application of quasi-application of quasi-solution methodsolution method
Resampling + Deringing:Resampling + Deringing:ResultsResults
Source image, upsampled by box filterLinear interpolationRegularization-based methodRegularization-based interpolation +quasi-solution deringing method
Image Enhancement by Image Enhancement by Regularization MethodsRegularization Methods
Introduction to regularizationIntroduction to regularization ApplicationsApplications
– Resampling (interpolation)Resampling (interpolation)– Deringing (Gibbs effect reduction)Deringing (Gibbs effect reduction)– Super-resolutionSuper-resolution
Super-Resolution:Super-Resolution:IntroductionIntroduction
The problem of super-resolution is to recover a The problem of super-resolution is to recover a high-resolution image from a set of several high-resolution image from a set of several degraded low-resolution images degraded low-resolution images
Super-resolution methodsSuper-resolution methods– Learning-basedLearning-based – single image super-resolution, – single image super-resolution,
learning database (matching between low- and high-learning database (matching between low- and high-resolution images) resolution images)
– Reconstruction-basedReconstruction-based – use only a set of low-resolution – use only a set of low-resolution images to construct high-resolution image images to construct high-resolution image
Super-Resolution:Super-Resolution:Inverse ProblemInverse Problem
The problem of super-resolution is posed as The problem of super-resolution is posed as error minimization problemerror minimization problem
z z – reconstructed high-resolution image– reconstructed high-resolution image vvkk – – kk-th low-resolution input image-th low-resolution input image
AAkk – downsampling operator, it includes – downsampling operator, it includes motion informationmotion information
Super-Resolution:Super-Resolution:Downsampling operatorDownsampling operator
AAkk – downsampling operator – downsampling operator
HHcamcam – camera lens blur (modeled by Gauss – camera lens blur (modeled by Gauss filter)filter)
HHatmatm – atmosphere turbulence effect (neglected) – atmosphere turbulence effect (neglected) nn – noise (ignored) – noise (ignored) FFkk – warping operator – motion deformation – warping operator – motion deformation
Super-Resolution:Super-Resolution:Warping operatorWarping operator
Warping operator Warping operator FFkk
Super-Resolution:Super-Resolution:RegularizationRegularization
The problem is ill-posed, and we use The problem is ill-posed, and we use Tikhonov regularization approach (same as in Tikhonov regularization approach (same as in resampling)resampling)
where ,where ,
Minimization by subgradient methodMinimization by subgradient method
Super-Resolution:Super-Resolution:ResultsResults
Linear methodPixel replication
Non-linear method
Super-resolutionFace super-resolution for the
factor of 4 and 10 input images
Source images
Super-Resolution:Super-Resolution:ResultsResults
The reconstruction of an image from a sequenceThe reconstruction of an image from a sequence
examples of input frames (of total 14)
linearly interpolated single frame
super-resolution result
Super-Resolution for VideoSuper-Resolution for Video
Super-Resolution
For every frame, we take current frame, 3 previous and 3 next frames. Then we process it by
super-resolution.
current frame
Super-Resolution:Super-Resolution:Results for VideoResults for Video
Nearest neighbor interpolation
Super-Resolution
Super-Resolution for video for a factor of 4
Super-Resolution:Super-Resolution:Results for VideoResults for Video
Bilinear interpolation Super-Resolution
Super-Resolution for video for a factor of 4
Super-Resolution:Super-Resolution:Results for VideoResults for Video
Bicubic interpolation Super-Resolution
Super-Resolution for video for a factor of 4
ConclusionConclusion
Increasing CPU and GPU power Increasing CPU and GPU power makes regularization methods makes regularization methods more and more important in more and more important in image processingimage processing
Regularization is a very powerful Regularization is a very powerful tool but each specific image tool but each specific image processing problem needs its own processing problem needs its own regularization methodregularization method
Thank you!Thank you!
http://imaging.cs.msu.ru/http://imaging.cs.msu.ru/