Post on 31-Jan-2018
transcript
1
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept. 1
Image Enhancement II:
Neighborhood Operations
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept. 2
Image Enhancement:Spatial Filtering Operation
• Idea: Use a “mask” to alter pixel values
according to local operation
• Aim: (De)-Emphasize some spatial frequencies
in the image.
2
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept. 3
Overview of Spatial Filtering
• Local linear oprations on an image
)1,1()1,1(
),1()1,1(),(
98
21
yxfwyxfw
yxfwyxfwyxg
Input: f(x,y), Output: g(x,y)
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept.
• We replace each pixel with a weighted
average of its neighborhood
• The weights are called the filter kernel
• What are the weights for the average of a
3x3 neighborhood?
Moving average
1 1 1
1 1 1
1 1 1
“box filter” Source: D. Lowe
3
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept. 5
Spatial Filtering: Blurring
• Example
1 1 1
1 1 1
1 1 1
1/9
Averaging Mask:
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept. 6
Image Enhancement:Spatial Filtering Operation
• Local linear oprations on an image
)1,1()1,1(
),1()1,1(),(
98
11
yxfwyxfw
yxfwyxfwyxg Input: f(x,y), Output: g(x,y):
Input Image
Mask
Output Image
Usually odd
4
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept.
Defining convolution
lk
lkglnkmfnmgf,
],[],[],)[(
f
• Let f be the image and g be the kernel. The
output of convolving f with g is denoted f * g.
Source: F. Durand
• Convention: kernel is “flipped”
• MATLAB: conv2 vs. filter2 (also imfilter)
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept.
Key properties
• Linearity: filter(f1 + f2 ) = filter(f1) + filter(f2)
• Shift invariance: same behavior
regardless of pixel location: filter(shift(f)) =
shift(filter(f))
• Theoretical result: any linear shift-invariant
operator can be represented as a
convolution
5
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept.
Important details • What is the size of the output?
• MATLAB: filter2(g, f, shape)
– shape = ‘full’: output size is sum of sizes of f and g
– shape = ‘same’: output size is same as f
– shape = ‘valid’: output size is difference of sizes of f and g
f
g g
g g
f
g g
g g
f
g g
g g
full same valid
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept. 10
Image Enhancement:Spatial Filtering Operation
• An important point: Edge Effects
– To compute all pixel values in the output image, we need to fill in a “border”
Mask dimension = 2M+1
Border dimension = M
6
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept. 11
Image Enhancement:Spatial Filtering Operation
• An important point: Edge Effects (Ex.: 5x5 Mask)
– How to fill in a “border”
• Zeros (Ringing)
• Replication (Better)
• Reflection (“Best”) a b
c d
a b
a b c d
a
a
c
c
b
b
d
d
• Procedure:
– Replicate row-wise
– Replicate column-wise
– Apply filtering
– Remove borders
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept.
Implementation
• What about near the edge?
– the filter window falls off the edge of the
image
– need to extrapolate
– methods (MATLAB):
• clip filter (black): imfilter(f, g, 0)
• wrap around: imfilter(f, g, ‘circular’)
• copy edge: imfilter(f, g, ‘replicate’)
• reflect across edge: imfilter(f, g, ‘symmetric’)
Source: S. Marschner
7
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept.
• What about near the edge?
– the filter window falls off the edge of the
image
– need to extrapolate
– methods:
• clip filter (black)
• wrap around
• copy edge
• reflect across edge
Source: S. Marschner
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept. 14
Image Enhancement:Spatial Filtering Operation 5x5 Blurring with 0-padding 5x5 Blurring with reflected padding
8
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept.
Examples of linear filters
0 0 0
0 1 0
0 0 0
Original
?
Source: D. Lowe
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept.
Practice with linear filters
0 0 0
0 1 0
0 0 0
Original Filtered
(no change)
Source: D. Lowe
9
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept.
Practice with linear filters
0 0 0
1 0 0
0 0 0
Original
?
Source: D. Lowe
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept.
Practice with linear filters
0 0 0
1 0 0
0 0 0
Original Shifted left
By 1 pixel
Source: D. Lowe
10
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept.
Practice with linear filters
Original
? 1 1 1
1 1 1
1 1 1
Source: D. Lowe
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept.
Practice with linear filters
Original
1 1 1
1 1 1
1 1 1
Blur (with a
box filter)
Source: D. Lowe
11
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept.
Practice with linear filters
Original
1 1 1
1 1 1
1 1 1
0 0 0
0 2 0
0 0 0
- ?
(Note that filter sums to 1)
Source: D. Lowe
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept.
Practice with linear filters
Original
1 1 1
1 1 1
1 1 1
0 0 0
0 2 0
0 0 0
-
Sharpening filter
- Accentuates differences
with local average
Source: D. Lowe
12
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept.
Sharpening
Source: D. Lowe
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept. 24
Examples of some other smoothing or “low-pass” filters:
00000
00000
11111
00000
00000
5
1
00100
00100
00100
00100
00100
5
1
10000
01000
00100
00010
00001
5
1
00001
00010
00100
01000
10000
5
1
13
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept.
Gaussian Kernel
• Constant factor at front makes volume sum to 1 (can be
ignored, as we should re-normalize weights to sum to 1 in any case)
0.003 0.013 0.022 0.013 0.003 0.013 0.059 0.097 0.059 0.013 0.022 0.097 0.159 0.097 0.022 0.013 0.059 0.097 0.059 0.013 0.003 0.013 0.022 0.013 0.003
5 x 5, = 1
Source: C. Rasmussen
fspecial(‘gauss’,5,1)
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept.
Choosing kernel width
• Gaussian filters have infinite support, but
discrete filters use finite kernels
Source: K. Grauman
14
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept.
Choosing kernel width
• Rule of thumb: set filter half-width to about
3 σ
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept.
Example: Smoothing with a Gaussian
15
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept.
Mean vs. Gaussian filtering
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept.
Gaussian filters
• Remove “high-frequency” components from the image (low-pass filter)
• Convolution with self is another Gaussian
– So can smooth with small-width kernel, repeat, and get same result as larger-width kernel would have
– Convolving two times with Gaussian kernel of width σ is same as convolving once with kernel of width σ√2
• Separable kernel
– Factors into product of two 1D Gaussians
16
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept.
Separability of the Gaussian filter
Source: D. Lowe
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept.
Sharpening revisited • What does blurring take away?
original smoothed (5x5)
–
detail
=
sharpened
=
Let’s add it back:
original detail
+ α
17
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept. 33
More on Linear Operations: Sharpening Filters
• Sharpening filters use masks that typically have
+ and – numbers in them.
• They are useful for highlighting or enhancing
details and high-frequency information (e.g.
edges)
• They can (and often are) based on derivative-
type operations in the image (whereas
smoothing operations were based on “integral”
type operations)
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept.
Differentiation and convolution
• Recall, for 2D function,
f(x,y):
• This is linear and shift
invariant, so must be
the result of a
convolution.
• We could approximate
this as
• which is obviously a
convolution with kernel
f
x lim
0
f x , y
f x,y
f
xf xn1,y f xn , y
x
-1 1
Source: D. Forsyth, D. Lowe
18
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept. 35
Derivative-type Filters
)1,(),(2)1,(y
),1(),(2),1(x
),()1,(y
),(),1(x
2
2
2
2
yxfyxfyxff
yxfyxfyxff
yxfyxff
yxfyxff 11
1
1
121
1
2
1
Laplacian:
2
2
2
22
yx
fff 121
010
141
010
1
2
1
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept. 36
Variations of the Laplacian Filter
Laplacian:
2
2
2
22
yx
fff 121
010
141
010
1
2
1
Same response in row/column directions
101
040
101
001
020
100
100
020
001
Consider: Same response in
diagonal directions
Together:
111
181
111
010
141
010
101
040
101
“Isotropic” filter
19
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept. 37
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept. 38
Sharpening Using the Laplacian Filter
),( ),( ),( 2 yxfyxfAyxg
111
181
111
A
Boosting High
Frequencies
20
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept.
Gaussian Unsharp Mask Filter
Gaussian unit impulse
Laplacian of Gaussian
))1(()1()( gefgffgfff
image blurred
image
unit impulse
(identity)
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept.
Edge detection
• Goal: Identify sudden changes (discontinuities) in an image – Intuitively, most semantic
and shape information from the image can be encoded in the edges
– More compact than pixels
• Ideal: artist’s line drawing (but artist is also using object-level knowledge)
Source: D. Lowe
21
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept.
Characterizing edges
• An edge is a place of rapid change in the
image intensity function image
intensity function
(along horizontal scanline) first derivative
edges correspond to
extrema of derivative
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept.
The gradient points in the direction of most rapid increase in intensity
Image gradient • The gradient of an image:
•
The gradient direction is given by
Source: Steve Seitz
The edge strength is given by the gradient magnitude
• How does this direction relate to the direction of the edge?
22
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept. 43
Using the first derivative for enhancement:
yfxfyfxffyf
xff
2/122
• Sobel Edge Detector:
• Given image f apply :
101
202
101
121
000
121
xf
y
f
Compute Magnitudes and Add
Edge Image
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept. 44
Application of Sobel gradient:
Many other edge detecting filters exist. Which is best?
23
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept. 45
• Linear filters of the type we have seen (with all positive coefficients) will blur the image and reduce certain kinds of noise.
• Nonlinear smoothing filters can also be considered – Instead of computing a weighted average over the
masked area, perform an operation on the sorted list
of pixels in the area.
• Order statistic filters are useful for removing
certain “impulsive” types of noise.
Order-statistics Filters
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept. 46
Order-statistics Filters
9821 ffff
9821 wwww
Inner product
fwfwgT
i
ii
Linear Filter
9821 ffff
9821 zzzz
sort
9821 wwww
Inner product
zwzwgT
i
ii
Nonlinear (OS) Filter
Nonlinear operator
24
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept. 47
Examples of Order-statistics Filters
9821 ffff
9821 zzzz
sort
9821 wwww
Inner product
zwzwgT
i
ii
Nonlinear (OS) Filter
000010000
000000001
100000000
000
4
1
2
1
4
1000
Median Filter
Min Filter
Max Filter
Trimmed Mean Filter
EE 264: Image Processing and Reconstruction Peyman Milanfar
UCSC EE Dept. 48
• For Gaussian noise removal: Use linear smoothing filter
• For impulsive (Salt+Pepper, heavytailed,..): Use order statistic filter