2023-04-26

ELEC6111: Detection and Estimation Theory

Course Objective

The objective of this course is to introduce students to the fundamental concepts of detection and estimation theory. At the end of the semester, students should be able to cast a generic detection problem into a hypothesis testing framework and to find the optimal test for the given optimization criterion.

They should also be capable of finding optimal estimators for various signal parameters, derive their properties and assess their performance.

ELEC6111: Detection and Estimation Theory

Topics to be covered Detection Theory: Hypothesis testing: Likelihood Ratio Test, Bayes’ Criterion, Minimax Criterion, Neyman-

Pearson Criterion, Sufficient Statistics, Performance Evaluation. Multiple hypothesis testing. Composite hypothesis testing. Sequential detection Detection of known signals in white noise. Detection of known signals in coloured noise. Detection of signals with unknown parameters. Non-parametric detection. Estimation Theory: Bayesian parameter estimation. Non-Bayesian parameter estimation. Properties of estimators: sufficient statistics, bias, consistency, efficiency, Cramer-Rao

bounds. Linear Mean-Square Estimation. Waveform Estimation.

ELEC6111: Detection and Estimation Theory

Text:Vincent Poor, “An Introduction to Signal Detection and

Estimation: Second Edition,” Springer, 1994.References:

H. L. Van Trees, “Detection, Estimation, and Modulation Theory,” John Wiley & Sons, 1968.

J. M. Wozencraft, and I. M. Jacob, “Principles of Communication Engineering,” John Wiley & Sons, 1965.

Grading Scheme:Assignment: 5%Project (Turbo Decoder): 10%Midterm/Quiz: 25%Final: 60%

ELEC6111: Detection and Estimation Theory

Project: Turbo Coding.

Requirements: Literature Review, Simulation (using a programming language and not Packages

and Tool Boxes)References :

J. Haguenauer , E. Offer and L. Papke   "Iterative decoding of binary block and convolutional codes",  IEEE Trans. Inf. Theory,  vol. 42,  pp. 429 1996. 

M.R. Soleymani, Y. Gao and U. Vilaipornsawai, Turbo coding for satellite and wireless communications, Kluwer Academic Publishers, Boston, 2002 (on reserve at the library).

ELEC6111: Detection and Estimation Theory

IMPORTANT NOTICE: In order to pass the course, you should get at least 60%

(36 out of 60) in the final. The midterm and the Final exams will be open book. Only

one text book (any text book covering the material of the course) will be allowed in the exam. No non-authorized copy of the text will be allowed in the class, in the midterm or in the final.

Failing to write the midterm results in losing the 25% assigned to the midterm. In the case of medical emergency, a student may be given permission to re-write the midterm.

Assignments are very important for the understanding of the course material. Therefore, you are strongly encouraged to do them on time and with sufficient care.

ELEC6111: Detection and Estimation Theory

What do detection and Estimation involve?

Detection and Estimation deal with extraction of information from Information Bearing Signals (or Data).

The difference between the two is that in Detection we deal with discrete results (Decisions) while in Estimation we deal with real values.

ELEC6111: Detection and Estimation TheoryApplications of Detection Theory

Digital Communications.Radar.Pattern Recognition.Speech Recognition.Cognitive Sciences.Business.Biology.

ELEC6111: Detection and Estimation TheoryApplications of Estimation Theory

Communications.Adaptive Signal Processing.Audio Processing.Image Processing.Video Processing.Business.Biology.

ELEC6111: Detection and Estimation TheoryMathematical BackgroundA probability distribution is defined in terms

of A set Γ of outcomes.A set of subsets of Γ, say, Σ (the set of events)A measure (a function on sets) assigning a real

value P(s) to each . Σ is a σ-algebra, i.e., it is closed under

complementation and countable union of its members.

P(s) is non-negative and its sum (integral) over Σ is one.

s

ELEC6111: Detection and Estimation TheoryBayes Rule

P(A): a priori probability.P(A|B): a posteriori probability.

The aim is that based on the observation (B) find the actual cause (A).

A BP(B/A)

)()|()()|(

BPABPAPBAP

ELEC6111: Detection and Estimation TheoryDetection as Hypothesis TestingDetection can be viewed as a hypothesis testing problem

where we assume that the nature (system under consideration) is in one of several states. Assumption that the nature is n one of these states is a hypothesis.

With each state (hypothesis) is associated with one probability distribution (model).

Our aim is then to find a decision rule that maps our observations to these distributions (hypotheses) in an “optimal” way.

Based on our definition of optimality, we will have different decision criteria, e.g., Bayesian, Minimax and Neyman-Pearson.

ELEC6111: Detection and Estimation TheoryBayesian Hypothesis TestingTo complete the model, we need a cost function, i.e., what

is the cost of deciding that the hypothesis is true while, in fact the hypothesis has been at work. Let,

The expected (average) risk when is true is:

where is the probability of deciding , i.e., being in the region

when the hypothesis is true:

iH jH

.trueisHwhenHdecidingofCostC jiij

1,0, jH j

i

dyHypHyPP jjiij )|(|

),()()()( 1100 jjjji

ijijj PCPCPCR ijP

iHy

i

jH

ELEC6111: Detection and Estimation TheoryBayesian Hypothesis TestingTake the simple example of binary hypothesis testing.

Assume that we have two hypotheses and corresponding to distributions and , i.e., we have

versus

We are now looking for a decision rule partitioning the observation space Γ into two sets (acceptance region) and

(rejection region).

0H 1H 0P1P

00 ~: PYH

11 ~: PYH

0c01

0

1

0

1)(

yif

yify

)(y

ELEC6111: Detection and Estimation TheoryBayesian Hypothesis TestingThe risk average over all hypotheses is

where is the a priori probability of hypothesis .

So,

Let , then

1

0

)()(j

jjRr

1,0, jj

1

0101

1

00

1

01110

)()(

)()(1)(

jjjjj

jjj

jjjjjj

PCCC

PCPCr

jH

)|()( jj Hypyp

dyHypCCCrj

jjjjj

jj

1

])|()([)(1

001

1

00

ELEC6111: Detection and Estimation TheoryBayesian Hypothesis TestingThe risk is thus minimized if we choose to be:

Assuming making wrong decision is more costly than deciding correctly, i.e., , we can write as:

where,

)}.|()()|()(|{

0)|()(

010000101111

1

0011

HypCCHypCCy

HypCCy jj

jjj

1

0111 CC 1

}.)|(/)|(|{)}|()|(|{

01

011

HypHypyHypHypy

.)()(

11011

00100

CCCC

}.)|(/)|(|{)}|()|(|{

01

011

HypHypyHypHypy

ELEC6111: Detection and Estimation TheoryLikelihood Ratio TestDefining the Likelihood ratio:

the decision problem can be formulated as a likelihood- ratio test:

,)|()|()(0

1

HypHypyL

)(0

)(1)(

yLif

yLifyB

ELEC6111: Detection and Estimation TheoryProbability of ErrorAssuming that correct decision costs nothing and

erroneous decisions have the same cost, we get the following cost criterion:

With this cost assignment,

Note that the here the average risk is equal to the probability of error.

jiji

Cij 10

).()()( 011100 PPr

ELEC6111: Detection and Estimation TheoryA posteriori probabilitiesUsing Bayes rule:

where is the overall density given as:

The probabilities and are called a posteriori

probabilities. Bayes rule can be written as:

.)()|(

)|(ypHyp

yHP jjj

)|()|()( 1100 HypHypyp

)(yp

)|( 0 yHP )|( 1 yHP

)}.|()|()|()|(|{

101000

1110101

yHPCyHPCyHPCyHPCy

ELEC6111: Detection and Estimation TheoryA posteriori probabilitiesNote that here we are comparing the average cost of

deciding :

versus the average risk of deciding :

and decide in favour of the choice with minimum risk.For the case of uniform (probability of error) cost criterion, the

Bayes decision rule is:

This is called MAP (Maximum A posteriori Probability) decision rule.

1H

)|()|( 111010 yHPCyHPC

0H

)|()|( 101000 yHPCyHPC

)|()|(0)|()|(1

)(01

01

yHPyHPyHPyHP

yB

ELEC6111: Detection and Estimation TheoryExample (The Binary Channel)

Here and

So, the likelihood ratio is:

jyifjyif

Hypj

jj

1

)|(

0 0

1 1

01

1

11

0

}1,0{

.11

01

)|()|()(

0

1

0

1

0

1

yif

yif

HypHypyL

ELEC6111: Detection and Estimation TheoryExample (The Binary Channel)Assume that one observes if then it is

decided that a “one” was transmitted, else decision is made in favour of “zero”.

For the case of uniform costs and , we have,

and

or equivalently,

0y )1( 01

2/110

)1(0)1(1

)0(01

01

ifif

B

01

01

)1(0)1(1

)1(

ifif

B

01

01

)1(1)1(

)(

ifyify

yB

ELEC6111: Detection and Estimation TheoryExample (The Binary Channel)For a Binary Symmetric Channel (BSC), and we

have,

The minimum Bayes risk is:

10

2/112/1

)(

ifyify

yB

}.1,{min)( Br

ELEC6111: Detection and Estimation TheoryExample (Measurement with Gaussian Error)Assume that we measure a quantity taking one of the two

values or but our measurement is corrupted by a zero-mean

Gaussian noise with variance . The problem can be formulated as

versus

where,

0

12

),(~: 200 YH

),(~: 210 YH

22 2/)(

21)|(

ix

i eHyp

ELEC6111: Detection and Estimation TheoryExample (Measurement with Gaussian Error)The likelihood-ratio is,

The Bayes decision rule is,

0

2exp

2121

)|()|()(

10201

2/)(

2/)(

0

122

0

221

y

e

e

HypHypyL

y

y

otherwise

yifyB

02

exp1)(10

201

ELEC6111: Detection and Estimation TheoryExample (Measurement with Gaussian Error)

i.e., is either strictly increasing or decreasing with depending on

whether or . Therefore, instead of comparing with

the threshold , we can compare with another threshold . The

decision rule will be,

where

201 )()()( yL

dyydL

yifyif

yB 01

)(

)(yL y

01 01 )(yL y )(1 L

2)log( 10

01

2

ELEC6111: Detection and Estimation TheoryExample (Measurement with Gaussian Error)

Measurement in Gaussian Noise: Uniform cost and equally likely values

ELEC6111: Detection and Estimation TheoryExample (Measurement with Gaussian Error) The minimum Bayes risk can be computed using,

where

with and .

1

0101

1

00 )()()(

jjjjj

jjj PCCCr

12

)log(

02

)log(

)()|()( 1

jdd

Q

jdd

Q

QdyHypP jjj

x

u duexQ 2/2

21)( 2

01 d

ELEC6111: Detection and Estimation TheoryExample (Measurement with Gaussian Error)Assuming uniform cost and equally likely values, we get:

For binary modulation: and , so

)2/()( dQr

bE 01 202 N

)/2()( 0NEQBERr b