56
Computer Vision and Graphics (ee2031) Digital Image Processing II Dr John Collomosse [email protected] Centre for Vision, Speech and Signal Processing University of Surrey
Computer Vision and Graphics (ee2031) Digital Image Processing II
Dr John Collomosse [email protected]
Centre for Vision, Speech and Signal Processing University of Surrey
Learning Outcomes After attending this lecture, and doing the reading and labwork, you should be able to:
• Recommend appropriate filtering techniques to enhance images under different types of image noise.
• Describe the phenomenon of image edges, and implement common edge detection approaches using linear filtering.
• Describe the use of basic image histogram operators to reveal hidden visual detail within an image.
• Explain aliasing and it’s relationship to Nyquist’s limit. Be able to discuss the implications for 2D and 3D graphics.
• Describe image interpolation, and its link to linear filtering and convolution.
Credit: Some images in these slides from Noah Snavely (Cornell). David Lowe (Columbia). Steve Seitz (Washington). Various creative commons sources incl. Wikipedia.
Recall: Convolution
Consider a window containing a set of values.
For each pixel in the input:
1. Window values are multiplied with image beneath
2. The sum of these products is written to output image.
e.g. (0 x 1) + (4 x 1) + (0 x 1) +... = 4/9
0 4 0 0 0 0
0 0 0 0 0 0
0 0 0 3 0 0
0 0 0 0 0 0
0 2 0 0 0 0
0 0 0 0 0 0
0 1 0 0
0 0 0 0
0 1 0 0
0 0 0 0
1 1 1
1 1 1
1 1 1
1/9 x
Convolution
Input f(x,y) Output g(x,y)
Convolution in detail: 1D
Consider an input signal f(x),
Transformed by filter h(x),
Into an output signal g(x).
Impulse
Gaussian
* = f(x) h(x) g(x)
[k = ½ width of filter] Continuous Discrete
Convolution in detail: 2D (Discrete)
0 4 0 0 0 0
0 0 0 0 0 0
0 0 0 3 0 0
0 0 0 0 0 0
0 2 0 0 0 0
0 0 0 0 0 0
0 1 0 0
0 0 0 0
0 1 0 0
0 0 0 0
1 1 1
1 1 1
1 1 1
1/9 x = * F(i,j) H(i,j) G(i,j)
Consider an input signal F(i,j),
Transformed by filter H(i,j),
Into an output signal G(i,j).
Quick Quiz
2.Which low-pass filter produced fewer image artifacts?
0.11 0.11 0.11
0.11 0.11 0.11
0.11 0.11 0.11
0.06 0.13 0.06
0.13 0.24 0.13
0.06 0.13 0.06 3x3 box filter 3x3 Gaussian
1.What is the convolution theorem? Why is it useful?
3. What is the freq. domain representation of these filters?
5. Explain why one low-pass filter outperforms the other?
4. What are the implications of compact support in 2D DFT of these filters?
Noise reduction: An Alternative Linear low-pass filters (esp. Gaussian filters) are good at attenuating “Gaussian noise”
f(x,y) = I(x,y) + N(0,σ)
1 10 100 1000
image = signal + noise
Gaussian noise
= 1 pixel = 2 pixels = 5 pixels
Applying a linear low-pass (Gaussian) filter
Effectiveness of linear filter on Gaussian noise (good)
Noise reduction: An Alternative Another common type of noise: “Salt and pepper” noise, e.g.
f(x,y) = I(x,y) if 0 > p > 0.94
0 if 0.94 > p > 0.97
1 if 0.97 > p > 1
Original image Gaussian noise Salt and pepper noise (each pixel has some chance of being switched to zero or one)
Salt and Pepper noise
Applying a linear low-pass (Gaussian) filter
Effectiveness of linear filter on Salt and Pepper noise (bad)
Why?
= 1 pixel = 2 pixels = 5 pixels p = 10%
Salt and Pepper noise
Is this a linear filter? (Can we use convolution?)
Salt and pepper noise best attenuated using a Median Filter
= 1 pixel = 2 pixels = 5 pixels
3x3 window 5x5 window 7x7 window
p = 10%
Salt and Pepper noise - Median
Gaussian filter
Median filter
Edge Detection Linear filtering can be used to detect edges (high pass filter)
An edge is an intensity discontinuity in an image, caused by one or more physical factors:
depth discontinuity
surface color discontinuity
illumination discontinuity
surface normal discontinuity
Edge Detection Linear filtering can be used to detect edges (high pass filter)
An edge is an intensity discontinuity
Plotting intensity along this cross-section
This is a step-edge. Real image edges are “softer” than this.
Edge Detection Differentiating the image signal f(x,y), e.g. δf/δx
|δf/δx|
f(x)
f(x) Partial derivative
Edge Detection To compute derivative on a discrete signal we subtract neighbouring samples (finite difference)
f(x,y)
0 0 0
-‐1 0 1
0 0 0
-‐1 1 -‐1 1 -‐1 1
The obvious way to implement this in our 2D linear filter framework:
δf/δx
0 -‐1 0
0 0 0
0 1 0 δf/δy
δf/δx
Image Gradient Filtering f(x,y) independently to get δf/δx and δf/δy gives us a vector quantity for each pixel; the image gradient...
The gradient points in the direcIon of most rapid increase in intensity
The edge strength is given by the gradient magnitude:
The gradient (edge) direcIon is given by:
Image Gradient Example of processed image:
δf/δx δf/δy
f(x,y)
Working with Real Edges
Noisy input image
Working with Real Edges
f
h
f * h
To find edges, look for peaks in
Smoothed edge detection Both smoothing and differentiation are possible via convolution
Convolution is associative
So combine smoothing/differentiation into single filter.
Gaussian DerivaIve of Gaussian (x)
Gaussian Derivatives The two first-order Gaussian derivatives.
The Gaussian derivative differentiates the image in one direction (i.e. x or y) and smoothes in the orthogonal direction
x-‐direcIon y-‐direcIon
Sobel operator A common approximation to the first-order Gaussian Derivative that fits in a 3x3 window is the Sobel filter:
-‐1 0 1
-‐2 0 2
-‐1 0 1 δf/δx
-‐1 -‐2 -‐1
0 0 0
1 2 1 δf/δy
Sobel is frequently used in Computer Vision as a simple edge detector. A less common approximation is the Prewitt filter:
Which low-pass filter do you think Prewitt is composed with?
-‐1 0 1
-‐1 0 1
-‐1 0 1 δf/δx
-‐1 -‐1 -‐1
0 0 0
1 1 1 δf/δy
Sobel operator: Example
Sobel operator: Example
Smoothed edges are imprecisely localised
Global Thresholding A simple way to decide if a pixel is part of an edge or not is to threshold the edge strength field at a constant value.
This creates a “binary mask”. There are more sophisticated thresholding techniques available in the literature.
Image Histograms
Sometimes poor capture conditions can hide visual detail.
In such situations you can try manipulating the image histogram.
Image Histograms
Simple transfer functions
Original
Brightness
(+ 0.3)
Contrast
(x 1.3)
Gamma Correction
out= in γ
Original
Gamma
(0.2)
Correct for the non-linear response of a display/capture device
Localises contrast enhancement to part of histogram e.g. shadow
Gamma Correction
out= in γ Correct for the non-linear response of a display/capture device
Here, y=1/γ
Histogram Equalisation Distributing the area/energy evenly under the histogram often improves visibility of detail.
This increases the dynamic range of the image
As before, achieved via a monotonic transfer function
Histogram Equalisation
number of pixels with intensity k or less
pixels in image Number of intensity levels in image
Image Warping
How can we reduce the size (scale) this large image so it fits on the screen?
Image Sub-sampling
Throw away every other row and column to create a 1/2 size image
-‐ called image sub-‐sampling
1/4
1/8
We are ‘re-sampling’ the signal at a lower
sampling rate.
Image Warping
1/4 (2x zoom) 1/8 (4x zoom) 1/2
Zoom in on the smaller images and you see artifacts that weren’t present in the larger images.
Aliasing
“Wave” artifacts appear in the subsampled version where fine detail was present in the original. This is called aliasing.
Aliasing Similar problems occur in 3D computer graphics with distant geometry or fine scale texture:
Aliasing If you sample a signal at “too low” a rate, you will get aliasing.
High frequencies “alias” as lower frequency signals
You must sample a signal using a sampling rate at least twice the highest frequency present.
This minimal sampling rate is called the Nyquist rate
The maximum frequency that can be sampled at a given rate is referred to as the Nyquist limit (i.e. = 2x Nyquist rate)
If you sample below the Nyquist rate, aliasing will occur
Anti-Aliasing We can avoid aliasing when image warping, by filtering the signal to remove frequencies above the Nyquist limit.
G 1/4
G 1/8
Gaussian 1/2
Anti-Aliasing - Comparison Sub-sampling with Gaussian pre-filtering
G 1/4 G 1/8 Gaussian 1/2
Anti-Aliasing - Comparison Sub-sampling without Gaussian pre-filtering
blur
F0 H *
subsample blur subsample … F1
F1 H *
F2 F0
Gaussian pre-filtering
blur
F0 H *
subsample blur subsample … F1
F1 H *
F2 F0
{ Gaussian pyramid
Anti-Aliasing - 3D Graphics How might we anti-alias when rendering 3D graphics? (Consider the window operation of convolution / low-pass filtering)
R. Cook – SIGGRAPH ‘86
Distributed Ray Tracing
Anti-Aliasing - 3D Graphics Two solutions:
1. Distributed Ray Tracing (Cook, SIGGRAPH 86)
2. MIP Mapping (multus in parvum)
(Williams, SIGGRAPH 83)
Upsampling How can we increase the size of this image?
The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer,
Recall a digital image has compact support
• It is a discrete point-‐sampling of a conInuous funcIon • If we could somehow reconstruct the original funcIon, any new
image could be generated, at any resoluIon and scale
1 2 3 4 5
d = 1 in this example
Upsampling How can we increase the size of this image?
But we don’t know f, so we have to model in some way
Simplest model approximaIng f, is
to duplicate the closest pixel.
• “Nearest Neighbour” interpolaIon • Not really an “interpolaIon”
1 2 3 4 5
The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer,
Ideal scenario i.e. if we knew f
Bilinear interpolation Spatial implementation:
4 known pixels (Q) at integer f(x,y)
Need to know P?
Nearest-‐neighbor interpolaIon Bilinear interpolaIon
Upsampling How can we increase the size of this image?
1 2 3 4 5
The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer,
Ideal scenario i.e. if we knew f
Recall: Convolution
Consider an input signal f(x),
Transformed by filter h(x),
Into an output signal g(x).
Impulse
Gaussian
* = f(x) h(x) g(x)
[k = ½ width of filter] Continuous Discrete
Linear Interpolation How can we increase the size of this image?
1 2 3 4 5
The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer,
Ideal scenario i.e. if we knew f
1 2 3 4 5
The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer,
2.5
1 Using triangular or “tent” kernel
Bilinear interpolation The “tent” gives us a linear interpolation of surrounding values, weighting by distance.
1 2 3 4 5
The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer,
2.5
1 Using triangular or “tent” kernel
In 2D?
The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer,
performs linear interpolaIon
(tent funcIon) performs bilinear interpola,on
Image Interpolation
“Ideal” reconstrucIon
Nearest-‐neighbor interpolaIon
Linear interpolaIon
Gaussian reconstrucIon
Image Interpolation
Nearest-‐neighbor interpolaIon Bilinear interpolaIon Bicubic interpolaIon
Original image: x 10
Image Interpolation
Also used for general image warping
Summary After attending this lecture, and doing the reading and labwork, you should be able to:
• Recommend appropriate filtering techniques to enhance images under different types of image noise.
• Describe the phenomenon of image edges, and implement common edge detection approaches using linear filtering.
• Describe the use of basic image histogram operators to reveal hidden visual detail within an image.
• Explain aliasing and it’s relationship to Nyquist’s limit. Be able to discuss the implications for 2D and 3D graphics.
• Describe image interpolation and its link to linear filtering and convolution.
