Neighborhood Neighborhood OperationsOperations

ObjectivesObjectives• Why are neighborhoods important?• What is linear convolution?

– discrete– templates, masks or filters– algorithm mechanics– graphical interpretation

• Describe non-linear operators– maximum– minimum– median

• What is tiling?

Why are neighbourhoods important?Why are neighbourhoods important?

pixel

Because…Because…

• Provide context for individual pixels.

• Relationships between neighbors determine image features.

• Neighborhood operations:– noise reduction– edge enhancement– zooming

Noise reduction Edge Enhancement

Zooming

Neighbourhood OperationsNeighbourhood Operations

• Linear convolution (*)– A*B*C*D = B*C*D*A = ….

• Non-linear operators– median, max, min, ...

Convolution versus Spectral• We learnt two methods of processing images:

– Convolution– Spectral

• We analyzed and demonstrated how to build a processor (systolic, pipelined, parallel, cellular automaton) for 1D convolution.– 1D convolution is used in speech processing and in polynomial

multiplication.

• We will use visualized animations now to show in more detail how 2D convolution works for images.

• This should convince you how important it is to do convolution quickly in modern Spectral Architectures, especially for 3D etc.

2D Convolution2D Convolution

• Consists of filtering an image A using a filter (mask, template) B.

• Mask is a small image whose pixel values are called weights.

• Weights modify relationships between pixels.

We will show more examples of convolution now, especially for 2D data

A1,1 A1,2 A1,3 A1,4

A2,1 A2,2 A2,3 A2,4

A3,1 A3,2 A3,3 A3,4

A4,1 A4,2 A4,3 A4,4

B1,1 B1,2

B2,1 B2,2

BA C

C1,1 C1,2 C1,3

C3,1

C2,1 C2,2 C2,3

C3,2 C3,3

=

Filter,mask or template

Input image

ConvolvedImage

2 2

4 43 3

A1,1 A1,2

A2,1 A2,2

A1,3 A1,4

A2,3 A2,4

A3,1 A3,2 A3,3 A3,4

A4,1 A4,2 A4,3 A4,4

B1,1 B1,2

B2,1 B2,2 A1,1B1,1

A2,1B2,1

A1,2B1,

2

A2,2B2,

2

A1,1B1,1C1,1= A1,2B1,

2

A2,1B2,1 A2,2B2,

2

A1,1 A1,2

A2,1 A2,2

A1,3 A1,4

A2,3 A2,4

A3,1 A3,2 A3,3 A3,4

A4,1 A4,2 A4,3 A4,4

B1,1 B1,2

B2,1 B2,2A1,2B1,

1

A2,2B2,1

A1,3B1,

2

A2,3B2,

2

A1,2B1,1C1,2= A1,3B1,

2

A2,2B2,1 A2,3B2,

2

A1,1 A1,2

A2,1 A2,2

A1,3 A1,4

A2,3 A2,4

A3,1 A3,2 A3,3 A3,4

A4,1 A4,2 A4,3 A4,4

B1,1 B1,2

B2,1 B2,2A1,3B1,

1

A2,3B2,1

A1,4B1,

2

A2,4B2,

2

A1,3B1,1C1,3= A1,4B1,

2

A2,3B2,1 A2,4B2,

2

A1,1 A1,2

A2,1 A2,2

A1,3 A1,4

A2,3 A2,4

A3,1 A3,2 A3,3 A3,4

A4,1 A4,2 A4,3 A4,4

B1,1 B1,2

B2,1 B2,2

A2,1B1,

1

A3,1B2,1

A2,2B1,

2

A3,2B2,

2

A2,1B1,1C2,1= A2,2B1,

2

A3,1B2,1 A3,2B2,

2

A1,1 A1,2

A2,1 A2,2

A1,3 A1,4

A2,3 A2,4

A3,1 A3,2 A3,3 A3,4

A4,1 A4,2 A4,3 A4,4

B1,1 B1,2

B2,1 B2,2B1,1 B1,2

B2,1 B2,2B1,1 B1,2

B2,1 B2,2

Mathematical NotationMathematical Notation

1 1

1,1,,

Mk

ki

Nl

ljljkijilk BAC

NMB

A1,1B1,1C1,1= A1,2B1,

2

A2,1B2,1 A2,2B2,

2

ConvolutionConvolution

4 4 7 9

4 3 8 9

3 5 9 9

3 6 10 9

-1 2

-1 2

BA C

=

Filter,mask or template

Input image

ConvolvedImage

2 2

4 43 3

6

9

16

23

26

27

21

19

17

Convolution sizeConvolution sizeImage size = M1 N1

Mask size = M2 N2

Convolution size = M1- M2 +1 N1-N2+1

N1

N2

N1-N2+1

Typical Mask sizes= 33, 5 5, 77,9 9, 1111

What is the convolved image size for a 128 128 image and 7 7 mask?

*1 1 11 1 1 1 1 1

1 1 11 1 1 1 1 1

1 1 11 1 1 1 1 1

1 1 11 1 1 1 1 1

1 1 11 1 1 1 1 1

1 1 11 1 1 1 1 1

1 1 11 1 1 1 1 1

1 1 11 1 1 1 1 1

1 1 11 1 1 1 1 1

=

We convolve with 9*9 averaging filter

Nonlinear Neighbourhood Nonlinear Neighbourhood OperationsOperations

• Maximum

• Minimum

• Median

1,1,

1:,1:,

1,1,1:,1:

,

1,1,1:,1:

,

median

min

max

ljkijiNlljMkki

lk

ljkijiNlljMkki

lk

ljkijiNlljMkki

lk

BAC

BAC

BAC

We discussed already sorter architecture (three variants – pipelined, butterfly combinational and sequential controller). It can be used for all these operations, and also for other non-linear operators

61 62

57 60

59 65

63 56

59 55 58 57

49 53 55 45

1 1

1 1C1,2=

62

60

59

636359

Max and Min OperationsMax and Min Operations

63=max, 59=min

61 62

57 60

59 65

63 56

59 55 58 57

49 53 55 45

C1,2=

Median OperationMedian Operation

1 1

1 1

1

1

1 1 1

62

60

59

63

65

56

55 58 57

62

59

65

60

63

56

57

58

55

987654321

59

rank

9x9 Median

Edge Detection

• What do we mean by edge detection?

• What is an edge?

What is Edge Detection?What is Edge Detection?

• Detects large intensity transitions between pixels

• Redraws the image with only the edges showing

0 0 0 33

0 0 45 78

0 45 23 33

0 0 42 76

0 0 0 38

What is an Edge?

Edge easy to findEdge easy to find

What is an Edge?

Where is edge? Single pixel wide or multiple pixels?Where is edge? Single pixel wide or multiple pixels?

What is an Edge?

Noise: have to distinguish noise from actual edgeNoise: have to distinguish noise from actual edge

Noise Noise is hereis here

What is an Edge?

Is this one edge or two?Is this one edge or two?

What is an Edge?

Texture discontinuityTexture discontinuity

Edge Detection – so what is an Edge Detection – so what is an edge to be detected?edge to be detected?

• What is an edge– A large change in image brightness of a short

spatial distance – Edge strength = (I(x,y)-I(x+dx,y))/dx

But this general definition still allows for many theories, software implementation and hardware architectures.

Now we will Now we will discuss and discuss and illustrate various illustrate various kinds of filter kinds of filter operatorsoperators

Edge Detection FiltersEdge Detection Filters

•High - Pass Filtering Eliminates Uniform Regions (Low Frequencies)

•edge “detection” or “enhancement”

Edge Detection FiltersEdge Detection Filters

Edge Detection ContinuedEdge Detection Continued

•Sum of Kernel Coefficients = 0

•differences in signs emphasize differences in pixel values

•reduces average image intensity

•Negative pixel values in output?

Edge Detection FiltersEdge Detection Filters

vertical

horizontal diagonal

Edge DirectionEdge Direction

Directional High Directional High Pass FiltersPass Filters

Convolution Edge Convolution Edge Detection using Detection using

Sobel and similar Sobel and similar operatorsoperators

Example of Sobel Operator

Sobel OperatorSobel Operator

Sobel Edge DetectionSobel Edge Detection

Convolution Application ExamplesConvolution Application Examples

--Edge DetectionWe apply Sobel Operator

-1 -2 -10 0 01 2 1

-1 0 1

-2 0 2

-1 0 1

Column Mask

Row Mask

as mask to a sub-field of a picture

-1 2 -1

0 0 0

1 2 1

p0, p1, p2

p3, p4, p5

p6, p7, p8 = (p6-p0)+2(p7-p1)+(p8-p2)

We can learn from the result obviously •The result of the above calculation for column mask is horizontal difference•With Row Mask we will get vertical difference

*

The final step of the convolution equation, dividing by the weight , must be ignored

--Edge Detection with Sobel Operator

The weight of a mask determines the grey level of the image after convolution.

Like the weight of Sobel Mask WW= (-1) + (-2) + (-1) + 0 + 0 + 0 +1 + 2 +1= 0

The resulting image lost its “lightness” to be dark.dark.

Convolution Application ExamplesConvolution Application Examples

Sobel OperatorSobel Operator

Sobel OperatorSobel Operator

-1 -2 -1 0 0 0 1 2 1

-1 0 1-2 0 2 -1 0 1

S1= S2 =

Edge Magnitude =

Edge Direction =

22

21 SS

2

11tanS

S

SobelPrewitt

Ticbetts

Canny

Comparison of Edge Detection Comparison of Edge Detection AlgorithmsAlgorithms

Edge DirectionEdge DirectionAssymetric kernels detect edges from Assymetric kernels detect edges from specific directionsspecific directions

EastEast

1 1 -1

1 -2 -1

1 1 -1

NorthEaNorthEastst

1 -1 -1

1 -2 -1

1 1 -1

NorthNorth

-1 -1 -1

1 -2 1

1 1 1

Robinson Robinson OperatorOperator

Robinson Compass MasksRobinson Compass Masks

-1 0 1-2 0 2 -1 0 1

0 1 2-1 0 1 -2 -1 0

1 2 1 0 0 0 -1 -2 -1

2 1 0 1 0 -1 0 -1 -2

1 0 -1 2 0 -2 1 1 -1

0 -1 -2-1 0 -1 2 1 0

-1 -2 -1 0 0 0 1 2 1

-2 -1 0-1 0 1 0 1 2

Arrows show edge directions

Roberts Roberts OperatorOperator

Roberts OperatorRoberts Operator

• Does not return any information about the orientation of the edge

22 ),1()1,()1,1(),( yxIyxIyxIyxI

),1()1,()1,1(),( yxIyxIyxIyxI

or

1 00 -1

0 1-1 0+

Prewitt OperatorPrewitt Operator-1 -1 -1 0 0 0 1 1 1

-1 0 1-1 0 1 -1 0 1

P1= P2 =

Edge Magnitude =

Edge Direction =

22

21 PP

2

11tanP

P

Prewitt Row

Edge Detection FiltersEdge Detection Filters

0 1 2-1 0 1 -2 -1 0

1 2 1 0 0 0 -1 -2 -1Original

and filtered cow

Edge Detection Edge Detection (continued)(continued)

• First Order (Gradient) Kernels

• Prewitt RowPrewitt Row

1 0 -11 0 -1

1 0 -11 0 -1

1 0 -11 0 -1

• Sobel RowSobel Row

1 0 -11 0 -1

2 0 -22 0 -2

1 0 -11 0 -1

• Combine Row and Column Operators

Edge Detection Filters: compare Edge Detection Filters: compare Prewitt Prewitt and Sobeland Sobel

1D Laplacian Operator

)(xf

x

xf

)(

2

22 )(

x

xfxf

first derivative

second derivative

2D Laplacian Operator2D Laplacian Operator

2

2

2

22 ,,

),(y

yxf

x

yxfyxf

0 -1 0-1 4 -1 0 -1 0

1 -2 1-2 4 -2 1 -2 1

-1 -1 -1-1 8 -1-1 -1 -1

Convolution masks approximating a Laplacian

This is just one example of Laplacian, we can use much larger window

0 -1 0-1 4 -1 0 -1 0

Input Mask Output

Image Processing Operations for Early Vision:

Edge Edge DetectionDetection

Reminder: Effect of FiltersReminder: Effect of Filters

low

high

EdgesEdges… are the important part of images

intensity

color

edges

textures

contours

condensation...

simplest, least robust

most difficult, most robust There are many letters B occluded

by black shape here. How to find them?

Edge Detection

• Edges are curves in the image plane across which there is a “significant” change in image brightness.

• The goal of edge detection is the construction of an idealized line drawing

Image Processing OperationsImage Processing Operations

Pixels on edgesPixels on edges

Edges foundEdges found

Edge effects: rarely ideal edgesEdge effects: rarely ideal edges

Not all information is created equal...

Causes of edgesCauses of edges

• Depth discontinuity– One surface occludes another

• Surface orientation discontinuity– the edge of a block

• reflectance discontinuity– texture or color changes

• illumination discontinuity– shadows

Edges: causesEdges: causesWhat are they? Why?

four physical events that cause image edges...

What are they? Why?

discontinuities in

• surface color/intensity• surface normal• depth• lighting (specularities)

four physical events that cause image edges...

Edges: causesEdges: causes

Edges are image locations with a local maximum in image gradient in the direction of that gradient

(steepness)

Edges: causesEdges: causes

Formal Model of Edge (cont) Formal Model of Edge Formal Model of Edge

Formal Model of Edge (cont) Formal Model of Edge: Roberts Formal Model of Edge: Roberts

Formal Model of Edge (cont) Formal Model of Edge: Laplacian and Marr-Formal Model of Edge: Laplacian and Marr-Hildreth Hildreth

Formal Model of Edge (cont) Formal Model of Edge Formal Model of Edge

Formal Model of Edge (cont) Formal Model of Edge Formal Model of Edge

ThresholdsThresholds

very high thresholdoriginal image

Thresholds are important, done before or during edge detection.

Thresholds

very high thresholdoriginal image

ThresholdsThresholds

very high thresholdoriginal image

ThresholdsThresholds

very high threshold

reasonable

original image

ThresholdsThresholds

very high threshold

too low !reasonablethis all takes time...

original image

ThresholdsThresholds