Image Processing

Image Processing

• Goal – Start simple: look at small windows

– Identify useful image structures (‘Clues’ useful for recognizing objects)

– Eliminate irrelevant aspects of image appearance (neglect appearance variations that don’t help to explain object identity)

• First step (usually) in any algorithm

• We don’t “see” most information in image


+

Can add invisible change


+

=Can add invisible change

• Important information small fraction of totalInitial Image

• Important information small fraction of totalInitial Image Important information

Boundary locations boundary sharpnesssize/direction of brightness change

• Important information small fraction of totalInitial Image Important information

Boundary locations edge sharpness;size/direction of brightness change

Reconstruct good semblanceof original image justusing info at boundaries

Reconstruction

Elder: IJCV 34(2/3), 97–122 (1999)

Image Processing

• Goal– Small windows

– Identify useful image structures (`clues’)

– Eliminate irrelevant aspects

• Example 1– Emphasize signal; suppress noise

Image Processing

• Goal– Small windows

– Identify useful image structures (`clues’)

– Eliminate irrelevant aspects

• Example 2– Detect “boundaries” (jumps in brightness)– De-emphasize slow variations in brightness

Why look at small windows?

• Local redundancy

– Neighboring image points strongly related (points on same object have similar color, texture)

Don’t need to know brightness at every pixel

Isolate just useful information (eg, average brightness in window)

What’s behindthe patch?

Why look at small windows?

• Details are important!

Boundary signalspresence of some object

Note• All image locations are equal

– Mannequin equally likely to be anywhere in image

Note• All image locations are equal*

– Mannequin equally likely to be anywhere in image

*Not quite true for photographs (people center and compose images)

Conclusion• Analyze image over every small window

...

...

...


• Easiest: linear weighted sum of brightness in each window

...

...

...


• Easiest: linear weighted sum of brightness in each window Convolution (filtering)

...

...

...

What is image filtering?

For each pixel, modify value based on values of pixels nearby

Linear Filtering• New pixel value = weighted sum of nearby pixels

• Uses:

– Integrate information over regions

– Clean up noisy images

– Analyze image at different resolutions

– Detect image patterns, brightness boundaries

– Connects to Fourier analysis

Example (animated)

• 1D “image”

Average Sum nearby pixels with weights

I=(1 2 3 2 3 2 1)

1 1 1

3 3 3W=( )

7 8 7

3 3 3 I' = (2 2 )

73

73

832 2

Filtered Image I'

1 1 1

3 3 3

1 1 1

3 3 3

1 1 1

3 3 3

1 1 1

3 3 3

1 1 1

3 3 3

Averaging makes filtered curve smoother

Called mask, kernel, filter…

2D Linear Filtering

1 1 1111111

19

3 x 3 filter(averaging or box filter)

ImageNote: computer vision filtersusually have small number of pixels

Image

2D Linear Filtering

Filtered Image

Image

2D Linear Filtering

Filtered Image

Image

2D Linear Filtering

Filtered Image

Image

2D Linear Filtering

Filtered Image

Image

2D Linear Filtering

Filtered Image

Image

2D Linear Filtering

Filtered Image

Image

2D Linear Filtering

Filtered Image

Image

2D Linear Filtering

Filtered Image

Filtering Equations (correlation!)

IOriginal image

filter(or mask, kernel…)

W

Filtered image W I


IOriginal image

filterW

Filtered image W I

is a “little image” containing the weights with which the pixels of are summedW I

ProcedureFor each filter position

• Multiply filter and image entries in corresponding positions• Sum and record result at position under filter center


IOriginal image

W

Filtered image W I

( , ) ( , ) ( , )N N

j N i N

W I x y W i j I x i y j

filter


IOriginal image

W

Filtered image W I

( , ) ( , ) ( , )N N

j N i N


filter

Index ranges give filter size, here (2N+1) x (2N+1)


IOriginal image

W

Filtered image W I

( , ) ( , ) ( , )N N

j N i N


Easiest: use odd sized filters, symmetric index range [–N,N] so filter center at (0,0)

Index ranges give filter size, here (2N+1) x (2N+1)

filter

One Dimension (correlation!)

( ) ( ) ( )N

i N

W I x W i I x i

Filtered image

For filter of size (2N+1) centered on (0,0)

Convolution• Like Correlation with Filter Reversed

( ) ( ) ( )N

i N

W I x W i I x i

( , ) ( , ) ( , )N N

j N i N


1D

2D

( ) ( ) ( )N

i N

W I x W i I x i

( , ) ( , ) ( , )N N

j N i N



1D

2D

‘-’ instead of ‘+’crucial change!


• Many nice properties-a kind of multiplication

1D

2D

( ) ( ) ( )N

i N

W I x W i I x i

( , ) ( , ) ( , )N N

j N i N


Convolution

From now on, linear filtering equals convolution (unless I say otherwise)

Convolution procedure

IOriginal image

filterW

Convolved image *W I

ProcedureFor each filter position

• Flip (reflect) filter in both x and y directions

• Multiply filter and image entries in corresponding positions• Sum and record result at position under filter center

Convolution: symmetric form(doesn’t work for correlation)

1D ( ) ( ) ( )N

i N

W I x W i I x i


1D ( ) ( ) ( )N

i N

W I x W i I x i

{

j


1D ( ) ( ) ( )N

i N

W I x W i I x i

( ) ( ) ( )N i N

i jW I x

x

I

j i

W

{

j


1D ( ) ( ) ( )N

i N

W I x W i I x i

( ) ( ) ( )N i N

i jW I x

x

I

j i

W

Convention: extend filter. Assign W(i)=0 for out-of-range ( [ , ] )i i N N


1D ( ) ( ) ( )N

i N

W I x W i I x i

( ) ( ) ( )N i N

i jW I x

x

I

j i

W

( ) ( ) ( )i jW I x IiW

j x

Convention: extend filter. Assign W(i)=0 for out-of-range ( [ , ] )i i N N


1D This extends for any number of convolutions

( )* ( ) ( )i j k x

xW V I W i V j I k

(again, with convention that everything out of range is zero for W and V)


2D ( , ) ( , ) ( , )N N

i N j N


( , ) ( , ) ( ', ')' '

W I x y W i j I i ji i x j j y


2D ( , ) ( , ) ( , )N N

i N j N


'( , ) ( , ) ( , '' '

)y j jj

W I Wj

x i ii i x

Iy

Convolution like multiplication!

• Commutative

• Associative

• Distributive (linear)

* *F G G F

*( * ) * *F G H F G H

*( ) * *F G H F G F H


• Commutative * *F G G F

'

( ) ( ) ( ')i i x

x F i G iF G


• Commutative * *F G G F

'

( ) ( ) ( ')i i x

x F i G iF G

'

( ') ) *(i i x

xGG i F i F


• Associative

( ) (* ( ) )i j k x

x F i GF G j HH k

*( * ) * *F G H F G H

( )* xF G H

0

Convolution: More properties

• Shift invariance * * *shifted shiftedshiftedW I W I W I

0 -1 11-101-1

000*I

CompareW

-1

-1 1 001-101

000*I

shiftedW

( , ) ( 1, )* *shifted x y x yW I W I

vs

Convolution: More properties• Shift invariance Fourier Transform connection

Convolution equivalent* to multiplication after Fourier Transform!!

InverseFT FT FTK I K I

Some Examples

Filter mask

,W i j

1.0

Filtered image

j i

Some Examples

Filter mask

1.0

,W i j

Plot of mask(weights)

Filtered image

j i

Some Examples

Filter mask

1.0

j

,W i j

0,0 1.0

, 0 for 0 or 0

W

W i j i j

Filtered image

i

Some Examples

In 1D the plot of W would look like this.

Filter mask Filtered image

Filter mask

,W i j

j i

1.0

Filter mask

, ( , ) ( , )j i


,W i j

1.0

j i

Next example

Filter mask

,W i j

1.0Shifted from (0,0)

j i

Filter mask

,W i j

1.0

0,1 1

Otherwise, , 0

W

W i j

j i

Filter mask

,W i j

1.0

j i

Filter mask

,W i j

j i

19

New example

Filter mask

,W i j

j i

19

1 1 1111111

19

Filter mask

,W i j

j i

19

Averaging filter Blur

Original Filtered

(1D)

Filter mask

Original Filtered

(1D)

Filter mask

13

Original Filtered

(1D)

Filter mask

Original Filtered

(1D)

Filter mask

Original Filtered

(1D)

Filter mask

Note how the original sharp transition gets blurred

2.0

j i

1.0

j i

Filter mask

Warm up…

1

j i

Filter mask

2

j i

1

j i

-

Equivalent

2.0

j i

1.0

j i

Filter mask

Warm up…

2.0

j i j i

19

Filter mask

j i

19

Filter mask

2.0

j i j i

19

Filter mask

-

Filter mask

1.89

19

(a peak in a trough)

Original

in 1D

-

2.0

13

Filter mask

Original

in 1D

Filter mask

Original

in 1D

Filter mask

Original

in 1D

Filter mask

Original

in 1D

Filter mask

-1 0 110-110-1

16 ?

Different example

-1 0 110-110-1

16

Different example

1D example

-1 1

= ?o

1D example

o

-1 1

=

Only the jump survives!

A Detail:how to deal with border

Dealing with image border

What happens here?Trying to sum over pixels outside the image


Various choices:

1) Pad with zeros…

0

0

0

0

0

0

0

0

0

0


Various choices:

1) Pad with zeros2) Duplicate border


Various choices:

1) Pad with zeros2) Duplicate border


Various choices:

1) Pad with zeros2) Duplicate border3) Wrap (not so important in this class)


Various choices:

1) Pad with zeros 2) Duplicate border 3) Wrap 4) Or just crop (don’t try to extend original image; filtered image is smaller than original)

Image Filtered Image

Filtering to reduce noise

• Noise = what we don’t care about– Assume random noise added at each pixel

– Reduce noise by averaging over windows

• Random noise from different pixels tends to cancel

• Signal not much affected (image is redundant---pixel’s neighbors have similar brightness)

Simple Additive Noise

Simple Additive Noise• (Image = signal + random noise)

• Assume– No dependence of noise size on signal– Expected value of noise is zero– Noise added at each pixel independently – Type of noise is the same at all pixels.

or, more precisely:

, identically distributed, independent for

( ) 0i

n n n ni j i j

E n

i i iI s n

Averaging filter to reduce noise

Averaging Filter: Definition

• Mask has positive weights summing to 1

• Replaces each pixel with weighted average over its neighborhood

• Example: BOX filter has all weights equal

111

11 1

11

11/9

Averaging several times...

Averaging several times...

Image gets smoother; noise in patches instead of speckles

For smooth signals, averaging doesn’t affect the signal much and improves Signal/Noise

• Example: Image = Constant + Noise

– Average image over n-pixel window

i iI C n

/ 2

/ 2

'

averaged noise

N

i j i jj N

I C w n

C

For smooth signals, averaging doesn’t affect the signal much and improves Signal/Noise

• Example: Image = Constant + Noise

– Average image over n-pixel window

i iI C n

/ 2

/ 2

'

averaged noise

N

i j i jj N

I C w n

C

Smaller because of cancellations

Averaging a noisy imageYou can copy this code to matlab (substitute your image for prowler.pgm)

I=imread('prowler.pgm');imagesc(I), figure(gcf ) % “,” separates commands , figure(gcf) brings current figure to front (gcf ==“get current figure”)colormap gray % shows image as black and whiteS= size(I); % size of image I

In=double(I)+randn(S(1),S(2))*50; % Adds noise; randn creats a matrix of the given size with Gaussian random entries N(0,1) % `double’ forces image I to type double

imagesc(In),figure(gcf)

Is= conv2(double(In),ones(3,3)/9);imagesc(Is),figure(gcf) % convolve image with box filter. ``ones” creates a matrix % of given size containing ones

Is= conv2(double(Is),ones(3,3)/9);imagesc(Is),figure(gcf) pause % Wait for keyboard input before going on

% what does averaging again do? REPEATIs= conv2(double(Is),ones(3,3)/9);imagesc(Is),figure(gcf) , pause Is= conv2(double(Is),ones(3,3)/9);imagesc(Is),figure(gcf) , pause Is= conv2(double(Is),ones(3,3)/9);imagesc(Is),figure(gcf) , pause close all %close all figure displays

Averaging reduces noisek=100; T=1000; % k= image length (for 1D noise image) % T = number of trials RandIm =randn(k,1); plot(RandIm), figure(1), title('1D Noise Image')AverageNoise = sum((1/k)*RandIm), pausefigure(2)hist(sum((1/k)*randn(k,T))); axis([-1 1 0 inf]);xlabel([int2str(T),' trials of noise averaged over ', int2str(k),' pixels']); title('histogram')

Averaging for a normal image...(no noise)

What else is averaging good for?

• Remember original goal of image processing

– Identify useful image structures (`clues to objects’)– Throw away less useful parts of image

What else is averaging good for?

• Remember original goal of image processing

– Identify useful image structures (`clues to objects’)– Throw away less useful parts of image

• After smoothing…– Each pixel contains average brightness in its neighborhood.– This may be all you need to know about the neighborhood

To save only important information, keep just one pixel per neighborhood.

Original

Smoothed with1 1 1

111111

19

Keeping only every other pixel (along x and y directions)

New image ¼ sizeAll important information preserved

Image Compression

Original

Subsampled (every 7th pixel)

Image Compression

Original

Subsampled (every 7th pixel)

Smoothed, then sampled

Averaging: problems

Example: Averaging/smoothing with Box Filter

Star Star “averaged” with Box


Star Star “averaged” with Box Not very smooth!



Artifact from sharp edgeof box filter

Improved averaging filter

Smoothing as Inference of Signal

• True signal is smooth.

• To infer a pixel’s “true” brightness without

noise, look at brightness of nearby pixels

Signal

+Noise

signal plus noise


• Infer brightness at central pixel only using nearby pixels

Signal

+Noise

signal plus noise

Mask matched to signal


• Infer brightness at central pixel only using nearby pixels

Adjust size of averaging mask (Match signal smoothness: reduce noise without oversmoothing signal)

Signal

+Noise

signal plus noise

Mask matched to signal


• Mask size should match signal smoothnessSignal

signal plus noise

Mask

+Noise


• mask size should match signal smoothness

• Closer pixels are more similar Similarity falls off smoothly with distance

Signal

+Noise

signal plus noise

mask


• mask size should match signal smoothness

• Closer pixels are more similar Similarity falls off smoothly with distance

Make mask weights decrease smoothly with distance from filter center

Signal

+Noise

signal plus noise

mask

From box filter to Gaussian

Averaging with Gaussian mask• Gaussian weights nearby pixels more

• Smooth roll off in weights

Gaussian Smoother (Rotationally Symmetric)

2 2

2 2

1( , ) exp2 2

x yG x y

Gaussian filter gives reasonable model of blurring (eg from lens)

Star

Smoothing with Gaussian

Blurry Star(Gaussian Smoothed)

Smoothing with Gaussian

Box SmoothingGaussian Smoothing

Artifact from sharpnessof box filter

No artifact

Gaussian Filter: useful property• Convolving 2 Gaussians yields* new Gaussian!

2 2

2 2

1, exp2 2

x yG x y

2 21 2 1 2*G G G

Recall Gaussian Definition:

2 21 2

2 2

2 2 2 21 2 1 2

1, exp2 2

x yG x y


2 2

2 2

1, exp2 2

x yG x y


2 21 2

2 2

2 2 2 21 2 1 2

1, exp2 2

x yG x y

New Gaussian is wider, with (more blur)2 2 21 2new

2 21 2 1 2*G G G


2 2

2 2

1, exp2 2

x yG x y


2 21 2

2 2

2 2 2 21 2 1 2

1, exp2 2

x yG x y


2 21 2 1 2*G G G

* Strictly true only for continuous functions, not discrete filters defined on pixels


2 2

2 2

1, exp2 2

x yG x y


2 21 2

2 2

2 2 2 21 2 1 2

1, exp2 2

x yG x y


2 21 2 1 2*G G G

* Strictly true only for continuous functions, not discrete filters defined on pixels

See later for equations

Gaussian Filter• Associativity of convolution implies

• Several filterings with small Gaussians equivalent to single filtering with larger Gaussian

1 2 1 2* * * **

comboG G I G G I G I

2 2 21 2combo

This generalizes…• Convolving two smoothing filters gives a

smoothing filter with more blur.

• Can get a lot of smoothing by convolving many times with a small (low blur) filter

This is faster to compute…

Gaussian Filter and Scale Space• Want to consider image at different resolutions

(levels of detail, different scales)

Smooth image with different sized smoothers

Branches,leaves Bushes Tree, ground, sky foreground, background





Human vision does this!(double face)




• Scale Space = same image viewed at different resolutions (size scales)





• Scale Space = same image viewed at different resolutions (size scales)

With Gaussians, only resolution matters…Whatever the sequence of Gaussian smoothing operations applied to an image, the result isthe same as smoothing once with a single Gaussian of the appropriate size/resolution.(With other filters, two different series of smoothing operations can give different results at the same resolution)


Gaussian Filter and Scale Space

• Scale Space (same image viewed at different resolutions)

• Many applications!– Fast image download (Transmit low resolution, progressively upgrade to higher

resolution)

– Image editing/blending

– Fast image searching (search image first at low resolution, refine at higher resolution)

– Image compression ….

Example: Image Blending

Implementation notes

• Gaussian has infinite size (G(x)>0 for all x)• For efficiency, cut off filter at large x. What x?• Rule of thumb: Use window with each side 5or 6

Implementation note 2• Gaussian and BOX filters are separable

, x yW x y w x w y


, x yW x y w x w y

2 2 2 2

2 2 2

1 1( , ) exp exp exp2 2 2 2 2

x y x yG x y


Can replace 2D convolution with two 1D ones:Much faster!

1. Convolve each row with 1D filter

2. Then convolve each column with 1D filter

Reduces computation by factor

, x yW x y w x w y

yw y

xw x

| |Wwidth of filter

Image

Separable Filtering (animated!)

Filtered Image

=

Image

Separable Filtering (animated!)

Filtered Image

=

Aside: convolution for continuous functions

( ) ( ) ( )N

i N

W I x W i I x i

W I x dy W y I x y

discrete

continuous

Aside: Continuous convolution

( ) ' ' ( ) ( )N

i N

W I x di W i I x i

1i i

W I x dy W y I x y

Discrete(closer analog)

continuous

2 2

2

1( , ) exp2 2

x yG x y

• Constants normalize the “weights” to 1, so Gaussian is an averaging filter.

( , ) 1dxdy G x y

• Above is for continuous functions. When you approximate Gaussian on grid of pixels, normalize so

,

( , ) 1aprxi j

G i j

Note: normalization

Convolving 2 Gaussians

• See next page for equations for 1D convolution

22

1 2 2 21 2 1 2

1 exp exp2 2

*2

x yyG G x dy

Equations (Gaussian convolution in 1D)

22

2 21 2 1 2

1 exp2 2 2

x yydy

22 2 2

2 2 21 1 22 1 2 2 2 2

2 1 2 12 2

1 2 1 2

1 exp2 2

y x xdy

Completing

squarein exponent

2 2 2 22 1

2 2 2 21 2 1 2 2 1

1 exp2

ex2 2

py xdy

21

2 22 1

xy y

Change variable

2

2 22 22 12 1

1 exp22

x

From normalizationof Gaussian

Summary• Filtering: analyze small image windows

• Linear filtering and correlation/convolution

– Uses• Smoothing/averaging/blurring• Sharpening• Boundary (edge) detection • Next set of slides: pattern detection

– Convolution: Technicalities• Symmetric form• Commutative, associative, etc.• Dealing with boundary

Summary

• Smoothing– Uses

• Noise reduction (not so important for this class)• Compression• Scale space: analyze image at different resolutions (see later)• Edge detection (see later)

– Technicalities• Gaussians (special nice properties)• Normalize sum of weights to 1!• Size of Gaussian mask (note: smaller gives faster computation)• Separability• Blurring a little many times blurring a lot once

Date post:	24-Feb-2016
Category:	Documents
Upload:	gotzon
View:	51 times
Download:	0 times

Image Processing

Documents