8/9/2019 Adaptive and Statistical Signal Processing
1/21
Page 2
Adaptive and Statistical Signal ProcessingContents
Course No:
1. Random Variables and Vectors
2. Random Variables and Vectors, Intro to Estimation
3. MVU, Cramer Rao Lower Bound
4. Linear Models, Best Linear Unbiased Estimator
5. Least Squares and Maximum Likelihood Estimation
6. Introduction to Bayes Estimation7. Linear Bayes Estimation
8. Linear Bayes Estimation, Stochastic Processes
9. Stochastic Processes10. Wiener Filter
11. Wiener Filter
12. Least Squares Filter
8/9/2019 Adaptive and Statistical Signal Processing
2/21
Page 3
Minimum Variance Unbiased Estimator (MVU)
Remember: Quality Criterion for estimator:
mean squared error: should be minimal
let then
with e and e 2 as the mean and variance of the estimation error e,respectively, e is also called the bias of the estimator.
For a minimum variance unbiased estimatior (MVU) our aim is to find
an estimator that has zero bias ( e =0), i.e. with minimumvariance e 2 of the error
Please note a MVU does not necessarily is the estimator with theminimum mean squared error, in fact there might be an estimatorthat is biased ( e 0) but has a lower mean squared error.However, from a practical point of view, MUVs are often easier tofind that estimators minimizing the mean squared error.
Often it is even impossible to find an MVU. In this case we even haveto use additional constraints, e.g. to find best linear unbiasedestimator (BLUE)
))(( 2 E
=e222 ))(( ee E =
=)( E
8/9/2019 Adaptive and Statistical Signal Processing
3/21
8/9/2019 Adaptive and Statistical Signal Processing
4/21
8/9/2019 Adaptive and Statistical Signal Processing
5/21Page 6
Cramer-Rao-Lower Bound
Provide an easy way to determine a lower bound on theestimator performance, i.e. a better estimator cannot exist
The MVU estimator does not necessarily attain the CRLB
Sometimes the best estimator is provided by the CRLB directly
8/9/2019 Adaptive and Statistical Signal Processing
6/21Page 7
Cramer-Rao-Lower Bound
The Cramer-Rao Lower Bound (CRLB) is given by 1)
This means that the variance of every unbiased estimator mustbe higher or equal than the CRLB
If the CRLB exists and one can write
then the MVU estimator is given byand has the variance
where I( x ) is the so called Fischer information:
i.e. then the MVU satisfies the CRLB with equality1) if the pdf p( x ; ) satisfies the regularity condition:
allfor0
);(ln =
x p E
=
2
2 );(ln(x)
x p E I
)( xg= )(1x I
8/9/2019 Adaptive and Statistical Signal Processing
7/21Page 8
Cramer-Rao-Lower Bound
cf. Information Theory: Information is log b p(x)
Intuitive explanation for
The Fischer information can be seen as the amount of information about that is available in the data
The more information is available for estimation, the lower theestimation variance!
Important properties of ln p( x ; )
non-negativity
additive for independent RV:
CLRB lowers when using additional (independent) random variables
)(1
)var(
I
=
=1
0
)];[(ln);(ln N
n
n x p p x
=
=
1
02
2
2
2 )];[(ln);(ln N
n
n x p E
p E
x
8/9/2019 Adaptive and Statistical Signal Processing
8/21Page 9
Efficient Estimators
An unbiased estimator is said to be efficient if it attains the CRLB:
it uses all the available data efficiently
An efficient estimator is always the MVU but the MVU is not necessarilyan efficient estimator
Taken from Kay: Fundamentals of Statistical Signal Processing, Vol 1: Estimation Theory, Prentice Hall, Upper Saddle River
8/9/2019 Adaptive and Statistical Signal Processing
9/21Page 10
Cramer-Rao-Lower Bound
Example: DC Level in white gaussian noise:
x[n] = A + w[n]
w[n] is WGN with variance 2
Taking the first derivative
with as the sample mean.
=
=
1
0
22
22
)][(2
1exp
)2(
1);(
N
n N An x A p
x
)(
)][(
1
)][(2
1
])2ln[(
);(ln
2
1
02
1
0
2
222
A x N
An x An x A A
A p N
n
N
n
N
=
=
=
=
=
x
=
==1
0
][1)( N
n
n x N
xg x
8/9/2019 Adaptive and Statistical Signal Processing
10/21
Page 11
Cramer-Rao-Lower Bound
Differentiating again gives:
This leads to the CLRB:
By using the result from the first derivative one obtains:
This means for DC level in WGN the sample mean is the MVUestimator!
22
2 );(ln
N A p =
x
N
A2
)var(
))()(()(2 Ag A I A x N = x
8/9/2019 Adaptive and Statistical Signal Processing
11/21
Page 12
General CRLB for Signals in White Noise
Assume a deterministic signal with an unknown parameter isobserved in WGN as:
The pdf of x depending on the parameter is given as:
Differentiating once produces:
and a second differentiation results in:
1,...,1,0][];[][ =+= N nnwnsn x
=
=
1
0
22
22
]);[][(
2
1exp
)2(
1);(
N
n
N nsn x p
x
=
=
1
022
2 ];[]);[][(
1);(ln N
n
nsnsn x
p
x
=
=
1
0
2
2
2
22
2 ];[];[]);[][(
1);(ln N
n
nsnsnsn x
p
x
8/9/2019 Adaptive and Statistical Signal Processing
12/21
Page 13
General CRLB for Signals in White Noise
Taking the expectation value results in:
This leads to the CRLB for signals in White Noise:
The form of the bound demonstrates the importance of thesignal dependence on .
Signals that change rapidly as the unknown parameter changesresult in accurate estimators
E.g. as we have seen with the DC level in WGN: s[n;]=
produces a CRLB of 2 /N.
21
0
2
];[)var(
=
N
n
ns
21
022
2 ];[1);(ln
=
=
N
n
ns p E
x
8/9/2019 Adaptive and Statistical Signal Processing
13/21
Page 14
Transformation of Parameters
Assume the we wish to estimate a parameter that is a functiong( ) of some more fundamental parameter and already knowthe CRLB for
Then the CRLB for g() can be obtained by (without proof):Let and be an estimator of
Then the CRLB for is
2
2
2
);(ln)var(
x p E
g
)( g=
8/9/2019 Adaptive and Statistical Signal Processing
14/21
Page 15
Transformation of Parameters
But: be carefull!
Non-linear transformations destroy the efficiency
e.g. DC Level in WGN, Estimator for A 2
Square of the sample mean: might be a resonable estimator
But is not even unbiased anymore
On the other hand: affine (linear) transformations
preserve efficiency!
2 x
22
222
)var()()( A N A x x E x E +=+=
bagg +== )()(
8/9/2019 Adaptive and Statistical Signal Processing
15/21
Page 16
CRLB for Vector Paramters
Commonly: vector of parameters =[ 1 , 2 ,., p ] T
Without proof the CRLB is found as the [i,i] element of theinverse of the Fischer Information Matrix I ( )
With the Fischer Information Matrix defined by:
A Fischer Information Matrix is always symmetric
In practice I ( ) is assumed to be positive definite and henceinvertible
[ ]iii )()var( 1 I
[ ] p j pi p E ji
ij ,...,1 and,...,1for);(ln
)(2
==
=
xI
8/9/2019 Adaptive and Statistical Signal Processing
16/21
Page 17
CRLB for Vector Paramters
More formally: The covariance matrix of any unbiasedestimator satisfies: 1)
Furthermore an unbiased estimator may be found thatattains the bound if and only if
Again this MVU estimator is then efficientHere p( x ) is a p dimensional function!
The CRLB represents a powerful tool to find efficient estimatorsfor vector parameters.
1) if the pdf p( x ; ) satisfies the regularity condition:
xallfor0
);(ln =
p E
0IC
)(1)
C )
))()(();(ln xI
x =
g p
)( x g=
8/9/2019 Adaptive and Statistical Signal Processing
17/21
Page 18
Example: Line Fitting
Consider the problem x[n] = A + Bn + w[n]
w[n] is again WGN
The parameter vector
from which the first derivatives follow as:
=
=
1
0
22
22
)][(2
1exp
)2(
1);(
N
n N Bn An x p
x
],[ B A=
n Bn An x B
p
Bn An x A
p
N
n
N
n
=
=
=
=
1
02
1
02
)][(1);(ln
)][(1);(ln
x
x
8/9/2019 Adaptive and Statistical Signal Processing
18/21
8/9/2019 Adaptive and Statistical Signal Processing
19/21
8/9/2019 Adaptive and Statistical Signal Processing
20/21
Page 21
Example: Line Fitting
The CRLB for B is lower than for A (for N>2)
The lower bound for estimation of B decrease with oder 1/N 3oposed to 1/N for A
B is easier to estimate than A Intuitive Explanation: changes of B are magnified by n
Finding Estimators for A and B:
The derivatives
can be rewritten as (after some manipulations) as
=
=
=
=
n Bn An x
Bn An x
B
p A
p p
N
n
N
n1
02
1
02
)][(1
)][(1
);(ln
);(ln);(ln
x
x
x
+
+
+
=
=
B B A A
N N N N
N N N N
N
B p A
p
p
)1(12
)1(6 )1(
6
)1(
)12(2
);(ln
);(ln
);(ln2
2 x
x
x
8/9/2019 Adaptive and Statistical Signal Processing
21/21
Page 22
Example: Line Fitting
with
This means for these estimators the CRLB is satisfied withequality, hence they are efficient MVU estimators.
=
=
=
=
+
+=
+
+
=
1
0
1
02
1
0
1
0
][)1(
12][
)1(6
][)1(
6][
)1()12(2
N
n
N
n
N
n
N
n
nnx N N
n x N N
B
nnx N N
n x N N N
A