A LOW-COMPLEXITY, MOTION-ROBUST, SPATIO-TEMPORALLY ADAPTIVE VIDEO
DE-NOISER WITH IN-LOOP NOISE ESTIMATION
Chirag Jain, Sriram SethuramanIttiam Systems (Pvt.) Ltd., Bangalore, India
Image Processing, 2008. ICIP
Outline
• Introduction• Proposed Algorithm– LLMMSE based spatial filtering– Temporal filtering– Noise estimation
• Experimental Results• Complexity Comparison• Conclusions and Future Work
Introduction
• Many of the successful techniques determine the weights needed to combine the neighborhood of the pixel being de-noised.– e.g., bi-lateral[2], anisotropic diffusions
method[3],spatial non local means (NL-means) de-noiser
[4].– Drawback: high computational complexity
[2] C. Tomasi, and R. Manduchi.,” Bilateral filtering for gray and color images,” Proceedings of the Sixth International Conference on Computer Vision, pp. 839-846, 1998.[3] P. Bourdon, B. Augereau, C. Olivier, and C. Chatellier, “Noise removal on color image sequences using coupled anisotropic diffusions and noise-robust motion detection,” EUSIPCO'04 -Signal Processing XII, Wien (Austria), pp. 1194 – Sep. 2004.[4] A. Buades, B. Coll, and J. M. Morel, “Denoising image sequences does not require motion estimation,” IEEE Int. Conf. on Advanced Video and Signal based Surveillance, 2005.
Introduction
• We propose a de-noising algorithm using the following principles:– Block-based noise estimation and de-noising;– Use of block motion vectors by a video encoder;– Locally adaptive Linear Minimum Mean Square Error
(LLMMSE) methods.– Reliance on Infinite Impulse Response (IIR) temporal
filtering.– Spatially adaptive weight selection for the spatial-
temporal IIR (ST-IIR) filter.
Proposed AlgorithmCase 1: stationary blocks
Case 3: well Motion Compensated blocks
Case 2: under-compensated blocks
LLMMSE based spatial filtering
• An 3x3 Gaussian filter is applied to obtain a output L (low pass filtered ).
• The output H = X - L (X is the noisy image).• Spatially filtered output X’(i,j) is computed as:
– σf2: variance of noisy signal(X) for every 8x8 block.
– σn2: the estimated noise variance of the sequence.
Temporal filtering
• Let X(n) be the current frame, Y(n-1) be the previous denoised frame.
• σfr_diff2 : the estimated variance of the
difference data in stationary areas
between X(n-1) and Y(n-2)• σblk_diff
2 : the variance of the difference between cur_blk and its co-located block in Y(n-1)
(prev_blk)
Temporal filtering• Is block stationary? If (2) is satisfied
and at least one of (3) or (4) are true:
– Case 1: stationary blocks (CL)• Is MC successful? If (5) is satisfied:
– Case 3: well Motion Compensated blocks(MC)• All remaining blocks belong to case 2:
– Case 2: under-compensated blocks
Temporal filtering
• Case 2 : – Blocks only spatially filtered
• Case 1&3 : – The ST-IIR filtering is performed at a pixel level by using
the CL or MC pixel in Y(n-1):
– pixel_diff : the difference between the current pixel and its CL or MC pixel in Y(n-1).
– For pixels with pixel_diff > σfr_diff(n-1), α = e^(-1/k4)
σfr_diff(n-1) : the deviation of stationary (n-1)
Noise estimation
• If the current frame X(n) has noise variance σn2 and
Y(n-1) has noise variance σp2 , the variance σfr_diff
2(n) (stationary)will be:
• Ideally, σp2 should be zero, but we should introduce
a confidence c for robustness:
– c: the average value of α used for ST-IIR.
Noise estimation
• If the Y(n-1) is not filtered, c = 1, σn2 = σfr_diff
2 . • The convergence of standard deviation of the
noise against frame number for two sequences:
Experimental Results
• Additive white Gaussian noise of a specified variance was added to clean video sequences.
• Encoded using a H.264 baseline profile encoder.
• NL-means[4]: window size of 21x21 and a similarity square neighborhood of 7x7 with (a = 1 and h = n) [4].
Experimental Results
Experimental Results
Complexity Comparison
• The proposed algorithm perform the decisions at a block level and the filtering at a pixel level.
• The proposed algorithm requires only 38 operations(+,-,*,look-ups) per pixel
• NL-means has a complexity of 441x49 weighted Euclidean distance calculations and exponential function lookups per pixel.
Conclusions and Future Work
• The proposed algorithm comes close to the performance of more complex algorithms such as the NL-means method.
• It is ideally suited for real time embedded implementations.
• The spatial filter’s output can be used by the encoder to avoid spurious motion vectors.