1

Statistics and Image Quality Evaluation III

Oleh Tretiak

Medical Imaging Systems

Fall, 2002

2

This Lecture

• Mean square error as quality measure– Shortcomings

• How to do ROC by hand

• Visual ROC experiment

• This and other files for todays lecture at– http://www.ece.drexel.edu/courses/ECE-S684/Notes/IQ3/

3

Analytic Image Fidelity

• Mean square difference between imagesu(m, n) - perfect image, u’(m, n) - defective

image

€

σ a2 =

1MN

u(m,n) − ′ u (m,n)( )2

n=1

N

∑m=1

M

∑

€

SNR =10 log10σ 2

σ a2

€

PSNR =10 log10(peak − to − peak value in reference image)2

σ a2

4

‘White Noise’ Pattern

5

Noise Patterns

• White, low frequency, and high frequency noises. All have same standard deviation

6

Effect of noise on image quaity: UL ~ original 8-bit image; UR ~ white noise; LL ~ low pass noise; LR ~ high pass noise. Noise standard deviation is equal to 8, so that PTP signal to rms noise ratio is 32.

7

Effect of noise on image quaity: UL ~ original 8-bit image; UR ~ white noise; LL ~ low pass noise; LR ~ high pass noise. Noise standard deviation is equal to 32, so that PTP signal to rms noise ratio is 8.

8

Conclusions

• Same SNR does not produce the same image degradation when noises are different

• ‘Low frequency’ noise is most visible

• ‘High frequency’ noise is most damaging

9

• “The mean square error criterion is not without limitations, especially when used as a global measure of image fidelity. The prime justification for its common use is the relative ease with which it can be handled mathematically…” Anil Jain, in Fundamentals of Digital Image Processing.

10

Method

• Rating on a 1 to 5 scale (5 is best)

• Rating performed by 21 subject

• Statistics:– Average, maximum, minimum, standard

deviation, standard error for Case A, Case B– Difference per viewer between Case A and

Case B, and above statistics on difference

11

Two Kinds of Errors

• In a decision task with two alternatives, there are two kinds of errors

• Suppose the alternatives are ‘healthy’ and ‘sick’– Type I error: say healthy if sick– Type II error: say sick if healthy

12

• X - observation, t - threshold

= Pr[X > t | H0] (Type I error)

= Pr[X < t | H1] (Type II error)

Choosing t, we can trade off between the two types of errors

H0

H1

a

b

Decision threshold

13

ROC Terminology

• ROC — receiver operating characteristic

• H0 — friend, negative; H1 — enemy, positive

• Pr[X > t | H0] = probability of false alarm = probability of false positive = PFP =

• Pr[X > t | H1] = probability of detection = probability of true positive = PTP =

14

The ROC

• The ROC shows the tradeoff between PFP

and PTP as the threshold is varied

H0

H1

Decision threshold

PFP

PTP

PTP

PFP

0 10

1

AZ

15

Binormal Model

• Negative: Normal, mean = 0, st. dev. = 1

• Negative: Normal, mean = a, st. dev. = b

fN(x)=12π

exp(−x2 / 2)

fP(x) =12π

exp[−(x−a)2 /2b2]€

PFP (t) =1−Φ(t)

€

PTP (t) =1−Φt − a

b

⎛ ⎝ ⎜

⎞ ⎠ ⎟

16

17

ROC from Experimental Data

-0.96 N-1.04 N-1.75 N1.85 N1.64 N0.70 N0.02 N

-0.49 N0.45 N

-1.75 N 0.888-1.04 N 0.777-0.96 N 0.666-0.49 N 0.5550.02 N 0.4440.45 N 0.3330.70 N 0.2221.64 N 0.1111.85 N 0

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

-2.00 -1.50 -1.00 -0.50 0.00 0.50 1.00 1.50 2.00 2.50

Strarting data: 9 samples of N(0, 1) pseudorandom variable, simulating negative population. Middle: data sorted, with cumulative distribution. Right: plot of middle.

18

ROC from Experimental Data 2

3.77 P1.18 P

-0.82 P3.05 P4.06 P

-1.01 P

-1.01 P 0.833-0.82 P 0.6661.18 P 0.4993.05 P 0.3323.77 P 0.1654.06 P 0

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

-2.00 -1.00 0.00 1.00 2.00 3.00 4.00 5.00

Strarting data: 7 samples of N(2, 1.5) pseudorandom variable, simulating positive population. Middle: data sorted, with cumulative distribution. Right: plot of middle.

19

ROC from Experimental Data 3

-1.746 N 0.888-1.035 N 0.777-1.011 P 0.833-0.965 N 0.666-0.817 P 0.666-0.494 N 0.5550.025 N 0.4440.448 N 0.3330.701 N 0.2221.183 P 0.4991.642 N 0.1111.848 N 03.049 P 0.3323.767 P 0.1654.059 P 0

FPF TPF1 1

0.888 10.777 10.777 0.8330.666 0.8330.666 0.6660.555 0.6660.444 0.6660.333 0.6660.222 0.6660.222 0.4990.111 0.499

0 0.4990 0.3320 0.1650 0

Formula in FPF column=IF(N4="N",O4,P3)

Formula in TPF column=IF(N4="P",O4,Q3)

0

0.2

0.4

0.6

0.8

1

1.2

0 0.2 0.4 0.6 0.8 1 1.2

Merged sorted data and ‘Empirical ROC’ plot.

20

Comparison of Finite and Infinite Sample

0

0.2

0.4

0.6

0.8

1

1.2

0 0.2 0.4 0.6 0.8 1 1.2

0

0.2

0.4

0.6

0.8

1

1.2

0 0.2 0.4 0.6 0.8 1 1.2

Empirial ROC9 negative and 7 positive cases

Theoretical ROCWhat would have been obtained from a

very large sample

21

Metz

• University of Chicago ROC project:

• http://www-radiology.uchicago.edu/krl/toppage11.htm

• Software for estimating Az, also sample st. dev. And confidence intervals.

• Versatile

22

Ordinal Dominance Theory

• Donald Bamber, Area above the Ordinal Dominance Graph and the Area below the Receiver Operating Characteristic Graph, J. of Math. Psych. 12: 387-415 (1975).

• Sample Az is an unbiased, efficient, asymptotically normal estimate of population Az

• Bamber gives a formula for variance of Az

• Worst-case estimate (overbound) of variance

σmax

2

=

A ( 1 − A )

min( NX

, NY

)

23

Visual ROC Target

• Target is 9x16 pixels, Gaussian noise plus 4x4 dark rectangle on one side

• Protocol: Answer L, R if you think target is on left or right.

• In this example, target is on left

24

Experiment Goal

• Test human (your) visual system processing power

• Target is easy to see in absence of noise• This sort of stimulus can be analyzed, and

theoretical best ROC found• Use paper form for your answers, enter your

results on computer & mail to me.• Alternate method: Download file ROC-test-

form.xls, enter data on form, and send to me by e-mail.

25

Group 1

26

Group 2