1ECE595 Digital Image Processing

Topics Covered: Matlab orientation and refreshment Review of linear algebra and linear systems Image transformations Image enhancement in spatial and frequency domains Image enhancement in spatial and frequency domains Image restoration and segmentation Advanced topics in medical applications

Image Processing Photoshop

Question: Any image processing function missing in Photoshop?

You are encouraged to collaborate on homeworkproblems and labs with your classmates. However,cheating of any kind, including copying of otherswork of any kind is prohibited, and will not betolerated All university school department rulestolerated. All university, school, department rulesapply.

Digital Image Processing Using MATLABby Gonzalez, Woods, and Eddins

Mastering MATLABby Hanselman and Littlefield

Digital Signal Processing

Numerical Analysis Mathematical Methods Basic statistics

Chapter 2: Digital Image Fundamentals

Chapter 2

Human Visual System Image sampling Image interpolation

A i f i d i Averaging for noise reduction Linear and Shift Invariant Systems

Fovea: 1.5mm x 1.5mmDensity of cones: 150,000 /mm2

170 m

Density of cones: 150,000 /mm

2Optical Nerve and Visual Cortex

Function of Eyes: Optical Electrical

Why do we need long optical nerves?

Optical Illusions

How accurately can we estimate and compare distances and areas? By which features can we detect and distinguish objects?

Adelson's checker shadow illusion

Square A is exactly the same shade of grey as square B.

Link

http://www.michaelbach.de/ot/

When interpreted as a 3D scene, our visual system immediately estimates a lighting vector and uses this to judge the property of the material.

The Brain Sees What We Don't

The human visual system is extremely powerful in recognizing objects, but is less well suited for accurate measurements of gray values, distances, and areasand areas.

Solution: Let computers do the math and measurements.

The brain studying the brain Image Sampling and Digitizing

Regular rectangular grid

Question: B and C are mathematically identical?

B CA

3Other regular sampling grids

Regular rectangular grid is one of the sampling grids.

Triangular grid Hexagonal grid

Irregular grids: require advanced math skills.

Example: Simulate cell growth.

Signal Sampling in 1D

Matrix Representation

Nj

Nj

ffffffff

F..................

......

......

222221

111211j

i

NNNjNN

iNijiiNN

ffff

ffffF

........................

......

21

21

Each pixel has one value only. Pixels are indexed by integers. No area is associated with a pixel.

Indexing in 2D and 3D

Image processing convention: start from upper left corner

Neighborhood Relations

4 and 8 neighbors

6, 18, 26 neighbors

Spatial Resolution

Question:

FOV=10mx20m

Same FOV. a: 3 4, b: 12 16, c: 48 64, d: 192 256When FOV is fixed, small matrix size low spatial resolution.

what is the resolution?

4Question: Spatial Resolution

Fovea: 1.5mm x 1.5mmDensity of cones: 150,000 /mm2

170 m

What is the spatial resolution in the vertical direction of the tree?

Clearly state your assumptions.

Intensity Resolution: Quantization

From 256 gray levels to 2 gray levels.In practice, intensity levels are sufficient for visualization in most situations.

Linear System and Shift Invariant System

X(n) T Y(n) T{X(n)} = Y(n)

Shift invariant: T{X(n-n0)} = Y(n-n0)

Question: Are they shift invariant operators?Square operator: Y(n) = T{X(n)} = X2(n)Flip operator: Y(n) = T{X(n)} = X(-n)

Linear Operator

X1(n) T Y1(n) T{X1(n)} = Y1(n)

X2(n) T Y2(n) T{X2(n)} = Y2(n)

Linear: T{aX1(n) + bX2(n)} = aY1(n) + bY2(n)

Question: Are they linear operators?Square operator: Y(n) = T{X(n)} = X2(n)Shift operator: Y(n) = T{X(n)} = X(n) + c

Homework: summation operator is linear.

Satisfy both requirements: Linear Shift Invariant System (LSI)

Brightness, Contrast and SNR

Brightness Intensity.

Contrast Dynamic range

SNR Signal to noise ratio

Question: High brightness but low contrast, examples?

Brightness, Contrast and SNR

Brightness Intensity. Contrast Dynamic rangeSNR Signal to noise ratioThe definitions of contrast and SNR may be different in different areas.

SNR in MRI and other medical imaging modalities:

)()(

backgroundstdsignalavgSNR

Question: which part is the background?

5Signal Averaging

One of the most widely used technique in medical imaging, especially in MRI.

Result: improve signal to noise ratio (noise reduction) Result: improve signal to noise ratio (noise reduction)

Question: Averaging changes numerator(signal), denominator(noise), or both?

)()(

backgroundstdsignalavgSNR

Signal Averaging

),(),(),( yxnyxfyxg

N

ii yxgN

yxg1

),(1),(

By averaging N noisy images, the averaged image will be

),( yxg

),(),( yxfyxg

noisesignal

noisesignal

N

N

1

1 22

The expectation value of

Variance and standard deviation of signal and noise will be

Signal Averaging

Requirements: Uncorrelated and zero mean noise Registered images

Questions: Signal averaging leads to higher contrast? Signal averaging leads to higher SNR?

Can you estimate the SNR improvement after averaging 100 noisy images?

Image interpolation

What is the intensity at k=1.2?

28 2710

16 14

3952

Inverse Problem

28 2710

16 14

3952

What is the original signal or image? Can we reliably reconstruct the original image/object

back from the sampled data we have?

What is the intensity at k=1.2?

Image Interpolation

Interpolation is model based. Optimized interpolation exists. Commonly used interpolation methods:

N t i hb Nearest neighbor Linear, bilinear, trilinear Cubic, bicubic, tricublic Spline, many variations

6Nearest neighbor

Linear

Order of continuous

Linear

Cubic

Spatial Resolution and Sampling Rate

Given resolution Sampling rate

Given sampling rate resolution

To fully recover the required details Nyquist criteria

Low sampling rate Less details

The lamp pole in c and d: partial pixel or partial volume effect

Question: The lamp pole in C is larger than the pole in D. Why?

Sampling vs Interpolation

Sampling means that all information is lost except at the grid points. (continuous discrete)

Interpolation is to fill the gap between discrete grids. (discrete continuous)(discrete continuous)

Read DIP Chapter 2

Basic Relationships between Pixels and Distance Measures

Arithmetic Operation Set and Logical Operations Set and Logical Operations

Geometric Transformation

Spatial transformation of coordinates Intensity interpolation

Affine Transformation Translate Rotate Scale Sheer

Affine Transformation

ScalingTranslation SheeringRotation

Rigid Body

7Affine Transformation in 3D

ScalingTranslation SheeringRotation

Rigid Body

Affine Transformation

2

1

2221

1211

2

1

tt

yx

aaaa

yx

01

2

1

2

1

2

1

1001

tytx

tt

yx

yxTranslate

00

)cos()sin()sin()cos(

2

1

yx

yx

Rotate

Affine Transformation

2

1

2221

1211

2

1

tt

yx

aaaa

yx

2

1

2

1

2

1

1001

tytx

tt

yx

yx

Translate

Scaling

222 10 tytyy

00

)cos()sin()sin()cos(

2

1

yx

yx

Rotate

Question: How to implement scaling: x ax, y by?

Affine Transformation

2

1

2221

1211

2

1

tt

yx

aaaa

yx

1

xsyysx

yx

ss

yx

y

X

y

X

11

2

1Sheer

Scale

ysxs

yx

ss

yx

y

X

y

X

00

2

1

Better Way: Homogeneous Coordinates

22221

11211

1

1

yx

taataa

yx

11001

Question: Write down the transformation of translation, rotation, scaling and sheering.

Whats the difference?

22221

11211

1

1

yx

taataa

yx

100

11

21

2221

1211

11

ttaaaa

yxyx

11001

8Advantages of Homogeneous Coordinates

What are the advantages of the 3x3 transformation matrix?

C t ti f ti Concatenating a sequence of operations

Reducing image blurring

Question: Why concatenation can reduce image blurring?

3D Affine Transformation

333323122322211131211

111

zyx

taaataaataaa

zyx

11000

333323111 ztaaaz

High Dimensional Affine Transformation?

F d i Inverse mapping

Implementation

Forward mapping Inverse mapping

Question: Which mapping is better? Why?

1. Forward mapping could lead holes and overlaps in the output image

2. Inverse mapping is more efficient to implement than forward mapping.

Implementation

Nearest Neighbor Linear Cubic

Affine Transformation for Registration

1. Implementation issues: matrix size and interpolation

2. How to determine the transformation matrix?

Marker Cost functions

Cross Correlation Mutual Information

Intensity Transformation

Gray level mapping

9Intensity Transformation

255

Transformation

Lookup Table

Input Output0 01 1

0 255

function T(r)

L = 256

1 12 1

180 221

255 255

Intensity Transformation

Image Negative

L-1

Purpose: Output image may be better suitable for visualization

0 L-1

s = L-1- r

Log Transformation

Purpose: Spreading/compressing a narrow range of the low intensity values in the input image into a wider range of intensities s =c*log(1+ r) c is a constant

Power Law Transformation

Purpose: the same as the log transformation

)c(rs transformation

Advantages: More flexible in comparison to log

transformation Varying one parameter is enough to

yield a change in the image Varying gamma can get a family of

transformation curves

)c(rs c = 1, = 1, 3, 4

Criteria: Subjective or Objective?

10

Which gray mapping method?

0 255

255

0 255

255

0 255

255

0 255

255

log =2negativethreshold

Piecewise-Linear Transformation Functions

Purpose: increase the dynamic range of intensities in the image

Determine 2 points (r1,s1) and (r2,s2) which band the transformation linetransformation line

r1, r2 relate to the input images1, s2 relate to the output image

Result is 3 sections each with a different linear transformation

Piecewise-Linear Transformation Functions

Application: Contrast Stretching

Disadvantage: requires considerably more user input.

Looks better after transformation?

Bit Plane Slicing Bit PlanesOriginal Least significant bit plane

Most significant bit planeFor pattern recognition, the most significant bit plane is enough

11

Recon from bit planes 7-8 bit planes 6-8 bit planes 5-8

Most medical images are saved in 16 bit integer, unsigned

Bit Planes (1-8) Bit Planes (9-16)

Original image in uint16 Recon image from bit plane 8 to 14

In most hospitals and universities, the first choice is keep everything.

Steganography

http://en.wikipedia.org/wiki/Steganography

Removing all but the two least significant bits of each color component produces an almost completely black image. Making that image 85 times brighter produces the image on the right side.

12

Histogram Processing

Histogram

Histograms are the basis for numerous spatial domain processing techniques

Simple to calculate easy to implement Simple to calculate, easy to implement

Histogram

What is a histogram? A discrete function h(rk) = nk where rk is the kth intensity value

and nk is the number of pixels in the image with intensity rk.

Normalized histogram p(rk) = nk/(total pixels), k = 0,1,2,,L-1 p(rk) ~ the probability of occurrence of intensity level rk in an

image. sum(p(rk)) = 1

Normalized Histogram

Assuming that we are using a normalized histogram s=T(r) and that 0

13

)()1()()1()(0

rpLdttpddL

drdT

dds

r

r

r

dsdrrpsp rs )()(

r r dttpLrTs 0 )()1()(

0drdrdr rr

The resulting ps(s) always is uniform, independently of the form of pr(r).

11

)()1(1)()()( LrpLrpds

drrpspr

rrs

Histogram Equalization

Histogram Equalization

Histogram of the input image Histogram of the output image

Histogram Equalization Implementation

1

)()1()(0

nMNL

rpLrTs

k

i

k

iirkk

1,...,2,1,00

Lk

MN i

Histogram Equalization

Input histogram Transformation function Output histogram

Light image

Dark image

Qualitative Quantitative

Low contrast

High contrast

Input imagesHistogram Equalized Histogram

Light image

Dark image

Low contrast

High contrast

14

http://en.wikipedia.org/wiki/Histogram_equalization

The original image before histogram equalization

http://en.wikipedia.org/wiki/Histogram_equalization

Same image after histogram equalization

Histogram Specification Local Histogram Processing

Fundamentals of Spatial Filtering Order Statistic Filter

Histogram Specification

Sometimes, histogram equalization is not the best approach

Generate a processed image with a specified histogram

Histogram Specification

pr(r): Input images PDF pz(z): specified or desired PDF for output

r dttprTs )()( r dttprTs 0 )()(sdttpzG

z

z 0 )()()()( rTzG

)]([)( 11 rTGsGz

Define:

Histogram Specification

Procedure Calculate T(r) -- input histogram Calculate G(z) desired histogram

Ob i G 1 f i f i Obtain G-1 -- transformation function

Input T(r) G-1 output

Problem: No analytical expression for T(r) and G-1 in most cases.

15

Histogram Specification

Discrete formulation

k

iirkk rprTs

0)()(

k

k

iizkk szpzGv

0)()(

)()]([ 11 kkk sGrTGz

Requirement: pz(zi) != 0

Why?

How to implement sk zk?

original equalized specifiedNote: Histogram specification is easier for discrete case, but the histogram of an output image is the approximation for desired histogram.

Local Histogram Processing

Define a mask and move the center from pixel to pixel for each neighborhood. (sliced window)

Calculate histogram equalization/specification function Map the intensity of pixel centered in neighborhood Map the intensity of pixel centered in neighborhood Can use new pixel values and previous histogram to

calculate next histogram (implementation details)

Example of Local Histogram Equalization

original global local

Fundamentals of Spatial Filtering

Spatial filtering basically consists ofp g y A neighborhood or a kernel A predefined operation

The value of the kernel center is the result of the filtering operation

16

Spatial Filtering Kernel

Operate on a neighborhood with an odd kernel size. E.g. 3x3, 5x5, etc.

)1,1()1,1(...),()0,0(...),1()0,1()1,1()1,1(),(

yxfwyxfwyxfwyxfwyxg

Simplified Notation

)1,1()1,1(...),()0,0(...),1()0,1()1,1()1,1(),(

yxfwyxfwyxfwyxfwyxg

zw TzwzwzwR

992211 ...

General Linear Filter

mnmn zwzwzwR ...2211 We proved that the weighted summation is aWe proved that the weighted summation is a

linear operation One to one correspondence between linear

spatial filters and filters in the frequency domain

Nonlinear spatial filter

Order Statistic Filter: Median Filter

Max, Min Filter Percentage Filter

Median Filter

Purpose: remove salt and pepper noise Operation: replace the centre pixel with the

neighborhood median A nonlinear filter based on the ranking of pixels in a A nonlinear filter, based on the ranking of pixels in a

kernel

8

5

2

6

7

4

8

5

9

8 2 5 5 4 6 7 8 9

2 4 5 5 6 7 8 8 9Sort

17

Other Percentage Filters

Use ranking to find image properties For 3x3 kernel, Max Filter (100%) - finds the bright spots in the image

R ( | k 1 9) R = max(zk | k=1,,9)

Min Filter (0%) - finds the darkest spots in the image R = min(zk | k=1,,9)

Predefined percentage

Averaging Filter

Purpose: remove high frequencies from the image (smoothing or blurring)

Operation: Assign the average value of a kernel to the center of a neighborhoodcenter of a neighborhood

The size of kernel controls the smoothing effect

8

5

2

6

7

4

8

5

9

8 2 5 5 4 6 7 8 9

Average: 6

Difference between median and average?

Averaging vs. Median Filter

X-Ray image 3x3 averaging filter 3x3 median filter

It seems that the averaging is quite close to the median. Why the output images are so different?

Weighted Average

1

1

1

1 1?

Gaussian Kernel Weighted Averaging Equation

1

1

1

1),(),(

),( s ttysxftsw

yxg

1

1

1

1

),(),(

s ttsw

yxg

You can design your own filters by modifying the weights.

18

1 x 1 3 x 3

Kernel Size

5 x 5

35 x 35

9 x 9

15 x 15

Image from Hubble Space Telescope

15 x 15 averaging kernel

Thresholding

Get a gross representation of objects Detect large objects

Image SharpeningImage Sharpening

Image Sharpening

Operators for sharpening: derivative operator which is proportional to the degree of intensity discontinuity.

Image differentiation enhances edges and other Image differentiation enhances edges and other discontinuities

How to compute the differentiation? Numerical Computing or Scientific Computing

Finite difference

Finite Difference

Finite difference approximates an operator(e.g., the derivative) and solves a problem on a set of points (the grid)p ( g )

FD: convergence, accuracy, speed, etc.

Finite Difference

xx-1 x+1

F d diff )()1( xfxff =

How to computexf

?

Forward difference: )()1( xfxfx

= -

)1()( xfxfxf

= -Backward difference:

)21()

21(

xfxfxf

= -Central difference:

19

Accuracy

Using Taylors Theorem to estimate the approximation errors:

Forward difference O(h) Forward difference O(h) Backward difference O(h) Central difference O(h2)

Finite Difference

xx-1 x+1

F d diff )()1( xfxff =

Question: Calculate the finite difference xf32 38 56

Forward difference: )()1( xfxfx

= -

)1()( xfxfxf

= -Backward difference:

)21()

21(

xfxfxf

= -Central difference:

Finite Difference

xx-1 x+1

2 f

How to compute2

2

xf

?

32 38 56

)(2)1()1(22

xfxfxfxf

= - 2x

Central difference:

Use the central difference formula for the first derivative and apply a central difference formula for the derivative of f' at x.

Question: Derive the formula and calculate

+

2

2

xf

DistanceVelocity

Acceleration

Observations

First-order derivative Horizontal segments zero Onset of step or ramp nonzero

R i h l Ramp with constant slope nonzero

Second-order derivative Horizontal segments zero Onset of step or ramp nonzero Ramp with constant slope zero

Laplacian Filter

Rosnefeld and Kak (1982) introduced Laplacian filter to image processing

Second derivative operator Isotropic operator: spatial invariantp p p

2

2

2

22

yf

xff

)(2)1()1(22

xfxfxfxf

)(2)1()1(22

yfyfyfyf

xx-1 x+1

y

y-1

y+1

20

Laplacian Filter

xx-1 x+1

y

y-1

y+1

),(4)1,()1,(

),1(),1(22

2

22

yxfyxfyxf

yxfyxfyf

xff

xx-1 x+1

This Laplacian filter implementation is isotropic in the x or y direction (90 degree isotropic)

Laplacian Filter

xx-1 x+1

y

y-1

y+1

),(4)1,1()1,1(

)1,1()1,1(22

2

22

yxfyxfyxf

yxfyxfyf

xff

xx-1 x+1

Add the effect of the diagonals to the filters to make it 45 degree isotropic

Laplacian Filter

Isotropic results for increments of 90 degrees

Isotropic results for increments of 45 degrees

Note: same operation as image smoothing

Alternatives:

Laplacian Filter: Image SharpeningSharpened Image = Original + Laplacian image

),(),(),(),(

),( 22

yxfyxfyxfyxf

yxg if Laplacian kernel center < 0if Laplacian kernel center > 0

Laplacian with scaling (Set minLaplacian without

scaling

scaling. (Set min to zero to raise all values above zero).

Use D4 sharpen kernel

Use D8 sharpen kernel with additional differentiation in the diagonal directions.

Unsharp Masking

Widely used in printing and publishing industries for image sharpening

Method: Method: Blur the original image Subtract the blurred image from the original. The result

difference image called the mask Add the mask to the original image

21

Unsharp Masking Highboost Filtering

),(),(),( yxfyxfyxgmask ),(),(),( yxgkyxfyxg mask

Original

Gaussian blurred

For weighting factor k,

k = 1, unsharp maskingk >1, highboost filtering

Mask

Unsharp mask, k=1

Highboost filtering, k>1

Point and Line Detection

Point Detection

Detect isolated points in an image Method:

A point at (x y) is detected if |R|>T A point, at (x,y), is detected if |R|>T T is a threshold R is the filter (R=w1z1+w2z2++w9z9)

Point Detection

)(4)1()1(

),1(),1(22

2

22

yxfyxfyxf

yxfyxfyf

xff

xx-1 x+1

y

y-1

y+1

),(4)1,()1,( yxfyxfyxf xx-1 x+1

Compute the Laplacian image Convolve with the image Threshold the image

Point Detection

22

Line Detection

Similar to point detection Sharpen an image Threshold the image to get lines

Line Detection

What are the problems of using the absolute value of Laplacian?

What is the limitation of the Laplacian in edge detection?p g

Double the width of lines Laplacian can detect thin

lines only.

Can Laplacian detect thicker lines by increasing the kernel size?

Line Detection

Detect lines in specified directions

Apply +45 degree line detector mask

Keep >0 values threshold

Gradient Filter 1st Order Derivatives

What is the gradient?

In 1D, gradient = slope

Gradient = 0.6

A

B

C

Question: from ABC.Gradient (AB) = 0.8,Gradient (BC) ??

Gradient in 1D

xfgxffgrad x

)()(

Gradient Filter

In 2D, The gradient of f at (x,y) is a two dimensional vector

f

yfxf

gg

yxffgrady

x),()(

23

The magnitude of the gradient:

22)),((yxggyxfmag

yx

The gradient is NOT isotropic, but the gradient magnitude is.

Approximation:

yxggyxfmag )),((

Pros and cons of the approximation?

The magnitude of the gradient:

22)),((yxggyxfmag

yx

The gradient is NOT isotropic, but the gradient magnitude is.

Approximation:

yxggyxfmag )),((

Pros and cons of the approximation?

Fast computing, but lost isotropic property (only for 90 rotation)

Gradient Filtering

Robert proposed a way to compute the gradient with cross differences:

6859

2/1268

259

68

59

])()[(zzzzfzzzzf

zzgzzg

y

x

Problems: 2 x 2 op matrix is awkward to implement

Gradient Filtering

Prewitt Operator

)()()()( 321987

zzzzzzgzzzzzzg x

)()(

)()(

)()(

741963

321987

741963

zzzzzz

zzzzzzf

zzzzzzg y

Gradient Filtering

Sobel Operator

)2()2()2()2( 321987

zzzzzzgzzzzzzg x

)2()2(

)2()2(

)2()2(

741963

321987

741963

zzzzzz

zzzzzzf

zzzzzzg y

Sobel has better noise suppression characteristics

Sensitivity to Noise

Th 1 t d 2 d d i tiThe 1st and 2nd derivative are sensitive to noiseUse smoothing to keep the main changes

24

Sensitivity to Noise

The 1st and 2nd derivative are sensitive to noise Use a blurring filter to eliminate details and thus only keep the main changes

R b t C

Sum of coefficients is 0. Why?

In an area of Roberts Cross Gradient Operator

Sobel Operator

constant intensity, the response should be 0.

Masks for detecting horizontal and vertical edges

Sobel operator is widely used in edge detection.

Contact lens

Tree leaves

Contact lens

Sobel |gx|

Sobel |gy|Sobel |gx|+|gy|

Masks for detecting diagonal edges

25

Sobel |gx|

S b l | | Sobel |gx|+|gy|

Smoothed with 5x5 averaging filter

Sobel |gy|Sobel |gx| |gy|

Sobel diagonal operators

Image Sobel gradient image

Sobel x- gradient image Sobel y-gradient image

Advance Techniques for Edge Detection

Marr-Hildreth edge detector Canny edge detector

Combining Spatial Enhancement Methods

=+

Laplacian

Sobel

5x5 averagesmoothing

Mask

Mask

Original

Gamma

+

Geometric, Intensity Transformation and spatial Filtering

Affine transformation Geometric modification

O ti t l ti t ti li d Operations: translation, rotation, scaling and shearing

Techniques: operation multiplication, interpolation

26

Geometric, Intensity Transformation and spatial Filtering

Intensity transformation Gray level mapping Operations: transformation function. Pixel by

i lpixel. Techniques: image negatives, threshold, log

transformation, gamma transformation, piecewise-linear transformation

Histogram processing: histogram equalization and specification, local histogram processing

Geometric, Intensity Transformation and spatial Filtering

Spatial filtering A neighborhood or a kernel. (Kernel level) A predefined operation Operations:

Smoothing: averaging, median Sharpening: Laplacian, Highboost, gradient Edge detection: Laplacian, gradient

Techniques Sliding window Finite difference