1
CSE 152, Spring 2019
Introduction to Computer Vision
Filtering
Introduction to Computer Vision CSE 152 Lecture 3
CSE 152, Spring 2019
Introduction to Computer Vision
Announcements • HW0 is due on wed
2
CSE 152, Spring 2019
Introduction to Computer Vision
Last Lecture
CSE 152, Spring 2019
Introduction to Computer Vision
Equation of Perspective Projection
Let f’>0 be the distance between O and Π’ • We have, by similar triangles, that for P=(x, y, z), the intersection of
OP with Π’ is (f’ x/z, f’ y/z, f’) • Establishing an image plane coordinate system at C’ aligned with i
and j, we get (x, y, z)→ ( f ' xz, f ' y
z)
3
CSE 152, Spring 2019
Introduction to Computer Vision
• Is the transform linear? (x, y, z)→ ( f ' xz, f ' y
z)
Answer: No. Because it divides by z.
• But we can make it linear by using homogeneous coordinates.
homogeneous image coordinates (2D)
homogeneous scene coordinates (3D)
• Converting from homogeneous coordinates
CSE 152, Spring 2019
Introduction to Computer Vision
The equation of perspective projection
Cartesian coordinates:
(x, y, z)→ ( f xz, f yz)
UVW
⎛
⎝
⎜⎜⎜
⎞
⎠
⎟⎟⎟=
f 0 0 00 f 0 00 0 1 0
⎛
⎝
⎜⎜⎜⎜
⎞
⎠
⎟⎟⎟⎟
XYZ1
⎛
⎝
⎜⎜⎜⎜
⎞
⎠
⎟⎟⎟⎟
Homogenous coordinates and camera matrix
f
4
CSE 152, Spring 2019
Introduction to Computer Vision
What if camera coordinate system differs from world coordinate system?
cP = bcR bP+ cOb
• Where cOb is the origin of the bird frame in the
world frame. • is the rotation between
the frames. bcR
{c}
P
{b}
BP1
⎡
⎣⎢⎢
⎤
⎦⎥⎥= A
BR BOA0T 1
⎡
⎣
⎢⎢
⎤
⎦
⎥⎥
AP1
⎡
⎣⎢⎢
⎤
⎦⎥⎥
CSE 152, Spring 2019
Introduction to Computer Vision
Full Camera Matrix
M = KΠwcT =
f s cx0 α f cy0 0 1
⎡
⎣
⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥
100
010
001
000
⎡
⎣
⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥
wcR cOw0T 1
⎡
⎣
⎢⎢
⎤
⎦
⎥⎥
Intrinsic Parameters
3 x 3
Projection 3 x 4
Extrinsic Parameters
4 x 4
q =M wP
• Mapping from point in homogenous world coordinates wP to homogenous pixel coordinates q
• Can then map q to Euclidean coordinates.
5
CSE 152, Spring 2019
Introduction to Computer Vision
Where are we going?
CSE 152, Spring 2019
Introduction to Computer Vision
Image Filtering: An example
Input Output
Filter
6
CSE 152, Spring 2019
Introduction to Computer Vision
(From Bill Freeman)
CSE 152, Spring 2019
Introduction to Computer Vision
Example: Smoothing by Averaging
7
CSE 152, Spring 2019
Introduction to Computer Vision
Why filter
JamesHays
CSE 152, Spring 2019
Introduction to Computer Vision
Image Filtering • Most common filters are linear filters and the process
of applying a linear filter is called convolution • Why filter
– Enhance images • Denoise, resize, increase contrast, etc.
– Extract information from images • Texture, edges, distinctive points, etc.
– Detect patterns • Template matching
– The “convolution” in Convolutional Neural Networks
8
CSE 152, Spring 2019
Introduction to Computer Vision
Convolution
Image (I)
Kernel (K)
*
Note: Typically the kernel is relatively small in vision applications.
-2
1 1 2
-1 -1
CSE 152, Spring 2019
Introduction to Computer Vision
Convolution: R= K*I
I R
∑ ∑−= −=
−−=2/
2/
2/
2/
),(),(),(m
mh
m
mkkjhiIkhKjiRKernel size
is m+1 by m+1
m=2
9
CSE 152, Spring 2019
Introduction to Computer Vision
Convolution: R= K*I
I R
∑ ∑−= −=
−−=2/
2/
2/
2/
),(),(),(m
mh
m
mkkjhiIkhKjiRKernel size
is m+1 by m+1
m=2
CSE 152, Spring 2019
Introduction to Computer Vision
Convolution: R= K*I
I R
∑ ∑−= −=
−−=2/
2/
2/
2/
),(),(),(m
mh
m
mkkjhiIkhKjiRKernel size
is m+1 by m+1
m=2
10
CSE 152, Spring 2019
Introduction to Computer Vision
Convolution: R= K*I
I R
∑ ∑−= −=
−−=2/
2/
2/
2/
),(),(),(m
mh
m
mkkjhiIkhKjiRKernel size
is m+1 by m+1
m=2
CSE 152, Spring 2019
Introduction to Computer Vision
Convolution: R= K*I
I R
∑ ∑−= −=
−−=2/
2/
2/
2/
),(),(),(m
mh
m
mkkjhiIkhKjiRKernel size
is m+1 by m+1
m=2
11
CSE 152, Spring 2019
Introduction to Computer Vision
Convolution: R= K*I
I R
∑ ∑−= −=
−−=2/
2/
2/
2/
),(),(),(m
mh
m
mkkjhiIkhKjiRKernel size
is m+1 by m+1
m=2
CSE 152, Spring 2019
Introduction to Computer Vision
Convolution: R= K*I
I R
∑ ∑−= −=
−−=2/
2/
2/
2/
),(),(),(m
mh
m
mkkjhiIkhKjiRKernel size
is m+1 by m+1
m=2
12
CSE 152, Spring 2019
Introduction to Computer Vision
Convolution: R= K*I
I R
∑ ∑−= −=
−−=2/
2/
2/
2/
),(),(),(m
mh
m
mkkjhiIkhKjiRKernel size
is m+1 by m+1
m=2
CSE 152, Spring 2019
Introduction to Computer Vision
Convolution: R= K*I
I R
∑ ∑−= −=
−−=2/
2/
2/
2/
),(),(),(m
mh
m
mkkjhiIkhKjiRKernel size
is m+1 by m+1
m=2
13
CSE 152, Spring 2019
Introduction to Computer Vision
Convolution: R= K*I
I R
∑ ∑−= −=
−−=2/
2/
2/
2/
),(),(),(m
mh
m
mkkjhiIkhKjiRKernel size
is m+1 by m+1
m=2
CSE 152, Spring 2019
Introduction to Computer Vision
Convolution: R= K*I
I R
∑ ∑−= −=
−−=2/
2/
2/
2/
),(),(),(m
mh
m
mkkjhiIkhKjiRKernel size
is m+1 by m+1
m=2
14
CSE 152, Spring 2019
Introduction to Computer Vision (Bill Freeman)
0 0 0
0 1 0
0 0 0
CSE 152, Spring 2019
Introduction to Computer Vision (Bill Freeman)
0 0 0
0 1 0
0 0 0
15
CSE 152, Spring 2019
Introduction to Computer Vision (Bill Freeman)
0 0 0
1 0 0
0 0 0
CSE 152, Spring 2019
Introduction to Computer Vision (Bill Freeman)
0 0 0
1 0 0
0 0 0
Shifted one Pixel to the left
Shift
16
CSE 152, Spring 2019
Introduction to Computer Vision (Bill Freeman)
1 1 1
1 1 1
1 1 1
CSE 152, Spring 2019
Introduction to Computer Vision (Bill Freeman)
1 1 1
1 1 1
1 1 1
17
CSE 152, Spring 2019
Introduction to Computer Vision
CSE 152, Spring 2019
Introduction to Computer Vision
Practice with linear filters
Original
1 1 1 1 1 1 1 1 1
0 0 0 0 2 0 0 0 0 - ?
Source: D. Lowe
18
CSE 152, Spring 2019
Introduction to Computer Vision
Practice with linear filters
Original
1 1 1 1 1 1 1 1 1
0 0 0 0 2 0 0 0 0 -
Sharpening filter - Accentuates differences with
local average
Source: D. Lowe
CSE 152, Spring 2019
Introduction to Computer Vision
19
CSE 152, Spring 2019
Introduction to Computer Vision
CSE 152, Spring 2019
Introduction to Computer Vision
Convolution and Correlation • 2d convolution
– Kernel is flipped over both axes
• 2d correlation – Kernel is not flipped
• When kernel is symmetric, convolution and correlation give the same result
∑ ∑−= −=
−−=2/
2/
2/
2/
),(),(),(m
mh
m
mkkjhiIkhKjiR
R(i, j) = K (h,k)I (i + h, j + k)k=−m/2
m/2
∑h=−m/2
m/2
∑
20
CSE 152, Spring 2019
Introduction to Computer Vision
Filters are templates • Applying a filter at
some point can be seen as taking a dot-product between the image and some vector
• Filtering the image is a set of dot products
• Insight – filters look like the
effects they are intended to find
– filters find effects they look like
KeypropertiesoflinearfiltersLet imfilter(I,f)beafunctionthatapplieskernalftoimageI
Linearity: imfilter(I, f1 + f2) = imfilter(I,f1) + imfilter(I,f2) Shiftinvariance:Samebehaviorregardlessofpixellocation
imfilter(I,shift(f)) = shift(imfilter(I,f))
Anylinear,shift-invariantoperatorcanberepresentedasaconvolution.
Source: S. Lazebnik
21
Convolutionproperties
Commutative:f*I=I*f– Conceptuallynodifferencebetweenfilterandimage
Associative:a*(b*c)=(a*b)*c– Oftenapplyseveralfiltersoneafteranother:(((I*f1)*f2)*f3)
– Thisisequivalenttoapplyingonefilter:I*(f1*f2*f3)
Correlationis_not_associativeSource: S. Lazebnik
Convolutionproperties(cont.)
• Distributesoveraddition:I*(f1+f2)=(I*f1)+(I*f2)
• Scalarsfactorout:– Forscalark
k(f*I)=(kf)*I=f*(kI)
• Identity:unitimpulsee=[0,0,1,0,0]I*e=I
Source: S. Lazebnik
22
CSE 152, Spring 2019
Introduction to Computer Vision
Properties of Continuous Convolution (Holds for discrete too)
Let f,g,h be images and * denote convolution
• Commutative: f*g=g*f • Associative: f*(g*h)=(f*g)*h • Linear: for scalars a & b and images f,g,h
(af+bg)*h=a(f*h)+b(g*h)
• Differentiation rule
xgfg
xfgf
x ∂
∂=
∂
∂=
∂
∂ **)*(
∫ ∫∞
∞−
∞
∞−
−−= dudvvugvyuxfyxgf ),(),(),(*
CSE 152, Spring 2019
Introduction to Computer Vision
Convolutional Neural Networks
• Core operation in CNN is, not surprisingly, convolution.
• During training of a CNN, the weights of the convolution kernels are learned.
• Can be extended to 3D – e.g., – image and R,G,B as channels (N x N x 3) – Volumetric data such as MRI, CT
23
CSE 152, Spring 2019
Introduction to Computer Vision
Filtering to reduce noise
• Noise is what we are not interested in. – Usually think of simple, low-level noise:
• Light fluctuations; Sensor noise; Quantization effects; Finite precision
– Complex noise: shadows; extraneous objects. • A pixel’s neighborhood contains
information about its intensity. • Averaging noise reduces its effect.
CSE 152, Spring 2019
Introduction to Computer Vision
Additive noise • I = S + N. Noise doesn’t depend on signal. • We’ll consider:
• Gaussian noise, ni drawn from Gaussian.
Ii = si + ni with E(ni ) = 0
si deterministic. ni a random var.
ni ,n j independent for i ≠ j
ni ,n j identically distributed
24
CSE 152, Spring 2019
Introduction to Computer Vision
Gaussian noise
Gaussian noise, ni drawn from Gaussian distribution with zero mean and standard deviation σ
CSE 152, Spring 2019
Introduction to Computer Vision
Gaussian Noise:sigma=1
Gaussian Noise: sigma=16
25
CSE 152, Spring 2019
Introduction to Computer Vision
Averaging Filter • Mask with positive
entries, that sum 1. • Replaces each pixel
with an average of its neighborhood.
• If all weights are equal, it is called a Box filter.
1 1 1
1
1 1 1 1
1
F
1/9
(Camps)
CSE 152, Spring 2019
Introduction to Computer Vision
Smoothing by Averaging Kernel:
26
CSE 152, Spring 2019
Introduction to Computer Vision
An Isotropic Gaussian • Circularly symmetric
Gaussian with variance σ2
CSE 152, Spring 2019
Introduction to Computer Vision
Smoothing with a Gaussian Kernel:
27
CSE 152, Spring 2019
Introduction to Computer Vision
Smoothing Gaussian filter Box filter
CSE 152, Spring 2019
Introduction to Computer Vision
An image with constant intensity + noise :
Each row shows smoothingwith gaussians of differentwidth; each column showsdifferent realizations of an image of gaussian noise.
The effects of smoothing
28
CSE 152, Spring 2019
Introduction to Computer Vision
Gaussian Smoothing
by Charles Allen Gillbert by Harmon & Julesz http://www.michaelbach.de/ot/cog_blureffects/index.html
CSE 152, Spring 2019
Introduction to Computer Vision
Gaussian Smoothing
http://www.michaelbach.de/ot/cog_blureffects/index.html
29
CSE 152, Spring 2019
Introduction to Computer Vision
Gaussian Smoothing
2=σ
8.2=σ 4=σ
original
CSE 152, Spring 2019
Introduction to Computer Vision
Efficient Implementation
• Both, the Box filter and the Gaussian filter are separable: – First convolve each row with a 1D filter – Then convolve each column with a 1D filter.
30
CSE 152, Spring 2019
Introduction to Computer Vision
Other Types of Noise • Impulsive noise
– randomly pick a pixel and randomly set to a value – saturated version is called salt and pepper noise
• Quantization effects – Often called noise although it is not statistical
• Unanticipated image structures – Also often called noise although it is a real repeatable
signal.
CSE 152, Spring 2019
Introduction to Computer Vision
Some other useful filtering techniques
• Median filter • Anisotropic diffusion
31
CSE 152, Spring 2019
Introduction to Computer Vision
Median filters : principle
Method : 1. rank-order neighborhood intensities 2. take middle value
• non-linear filter • no new gray levels emerge...
CSE 152, Spring 2019
Introduction to Computer Vision
Median filters: Example for window size of 3
The advantage of this type of filter is that it eliminates spikes (salt & pepper noise).
1,1,1,7,1,1,1,1 ↓
?,1,1,1,1,1,1,?
32
CSE 152, Spring 2019
Introduction to Computer Vision
Median filters : example
filters have width 5 :
CSE 152, Spring 2019
Introduction to Computer Vision
Median filters : analysis
median completely discards the spike, linear filter always responds to all aspects median filter preserves discontinuities, linear filter produces rounding-off effects Do not become all too optimistic
33
CSE 152, Spring 2019
Introduction to Computer Vision
Median filter : images
3 x 3 median filter :
sharpens edges, destroys edge cusps and protrusions
CSE 152, Spring 2019
Introduction to Computer Vision
Median filters : Gauss revisited Comparison with Gaussian :
e.g. upper lip smoother, eye better preserved
34
CSE 152, Spring 2019
Introduction to Computer Vision
Example of median
10 times 3 X 3 median
patchy effect important details lost (e.g. earring)
CSE 152, Spring 2019
Introduction to Computer Vision
35
CSE 152, Spring 2019
Introduction to Computer Vision
CSE 152, Spring 2019
Introduction to Computer Vision
Fourier Transform • 1-D transform (signal processing) • 2-D transform (image processing) • Consider 1-D
Time domain ßà Frequency Domain Real ßàComplex
• Consider time domain signal to be expressed as weighted sum of sinusoid. A sinusoid cos(ut+φ) is characterized by its phase φ and its frequency u
• The Fourier transform of the signal is a function giving the weights (and phase) as a function of frequency u.
36
CSE 152, Spring 2019
Introduction to Computer Vision
Fourier Transform • 1D example
– Sawtooth wave • Combination of harmonics
CSE 152, Spring 2019
Introduction to Computer Vision
Fourier TransformDiscrete Fourier Transform (DFT) of I[x,y]
Inverse DFT
x,y: spatial domainu,v: frequence domainImplemented via the “Fast Fourier Transform” algorithm (FFT)
37
CSE 152, Spring 2019
Introduction to Computer Vision
Fourier basis element Transform is sum of orthogonal basis functions Vector (u,v) • Magnitude gives frequency • Direction gives orientation.
€
e−i2π ux+vy( )
CSE 152, Spring 2019
Introduction to Computer Vision
Here u and v are larger than in the previous slide.
38
CSE 152, Spring 2019
Introduction to Computer Vision
And larger still...
CSE 152, Spring 2019
Introduction to Computer Vision
Using Fourier Representations
Limitations: not useful for local segmentation
Dominant Orientation
39
CSE 152, Spring 2019
Introduction to Computer Vision
Phase and Magnitude eiθ
= cosθ + i sin θ
• Fourier transform of a real function is complex – difficult to plot, visualize – instead, we can think of the phase and magnitude of the
transform
• Phase is the phase of the complex transform • Magnitude is the magnitude of the complex transform • Curious fact
– all natural images have about the same magnitude transform – hence, phase seems to matter, but magnitude largely doesn’t
• Demonstration – Take two pictures, swap the phase transforms, compute the
inverse - what does the result look like?
CSE 152, Spring 2019
Introduction to Computer Vision
40
CSE 152, Spring 2019
Introduction to Computer Vision
This is the magnitude transform of the cheetah pic
CSE 152, Spring 2019
Introduction to Computer Vision
This is the phase transform of the cheetah pic
41
CSE 152, Spring 2019
Introduction to Computer Vision
CSE 152, Spring 2019
Introduction to Computer Vision
This is the magnitude transform of the zebra pic
42
CSE 152, Spring 2019
Introduction to Computer Vision
This is the phase transform of the zebra pic
CSE 152, Spring 2019
Introduction to Computer Vision
Reconstruction with zebra phase, cheetah magnitude
43
CSE 152, Spring 2019
Introduction to Computer Vision
Reconstruction with cheetah phase, zebra magnitude
CSE 152, Spring 2019
Introduction to Computer Vision
The Fourier Transform and Convolution
• If H and G are images, and F(.) represents Fourier transform, then
• Thus, one way of thinking about the properties of a convolution is by thinking of how it modifies the frequencies of the image to which it is applied.
• In particular, if we look at the power spectrum, then we see that convolving image H by G attenuates frequencies where G has low power, and amplifies those which have high power.
• This is referred to as the Convolution Theorem
F(H*G) = F(H)F(G)
44
CSE 152, Spring 2019
Introduction to Computer Vision
Various Fourier Transform Pairs • Important facts
– scale function down ⇔ scale transform up i.e. high frequency = small details
– The Fourier transform of a Gaussian is a Gaussian.
compare to box function transform
CSE 152, Spring 2019
Introduction to Computer Vision
Next Lecture • Early vision
– Edges and corners • Reading:
– Chapter 5: Local Image Features