26
CRAMER-RAO INEQUALITY
CRAMER-RAO INEQUALITY
CRAMER-RAO INEQUALITY
Syllabus:
• Statement and proof of Cramer-Rao inequality
• Definition of minimum variance bound unbiased
estimator (MVBUE) of f().
• Proof of the following results .
• 1) If MVBUE exists for , then MVBUE exists
for f() , provided f is linear function.
• 2) If T is MVBUE for f() , then T is sufficient for
f() .
• If T is an unbiased estimator of satisfying
regularity condition, then T satisfies the
relation
CRAMER-RAO INEQUALITY
Syllabus ………..Continue
ninformatiofishertheisIwherenI
TVar )(;)(
1)(
• Examples and problems
Comparing estimators :
• Unbiased estimators can be compared via their variances and biased
estimators by comparing mean square errors. (Generally unbiased
estimators are preferable. )
• We can compare several unbiased estimators and find which one has
smallest variance, but this does not allow us to tell if an estimator has the
smallest variance amongst all unbiased estimators
• The Cramer-Rao lower bound provides a uniform lower bound
on the variance of all unbiased estimators of f = g( ).
• So if the variance of an unbiased estimator is equal to the
Cramer-Rao lower bound it must have minimum variance
amongst all unbiased estimators (so is said to be a minimum
variance unbiased estimator of f ).
CRAMER-RAO INEQUALITY : Statement
• Let X1, X2,….. Xn be a random sample from
f(x;), . Assume is a subset of the
real line. Let T=u(X1, X2,….. Xn ) be an
unbiased estimator of k(). Assume f(x;)
satisfies the regularity conditions, then
2
2
);(ln
)(
xfnE
kTVar
Regularity Conditions:
allandxallforexistsxfln );()1
n
nidxdxdxxf
1
21........;......)2
n
nidxdxdxxf
1
21........;......
n
nindxdxdxxfxxxt
1
2121........;),.....,(......)3
n
nindxdxdxxfxxxt
1
2121........;),.....,(......
inallforxfE
2
);(ln0)4
Proof : We have );().......;(1
nxfxf
2
2
1
1);();();();(
j
j
j
jxfxfxfxf
nj
jnxfxf );();(..........
2
22
21
11
1
);();();();(
1);();();(
);(
1
j
j
j
jxfxfxf
xfxfxfxf
xf
nj
jnn
n
xfxfxfxf
);();();();(
1..........
=
n
j
j
n
j
jxfxf
11
);();(ln
1
2
1
1);();(ln);();(ln
j
j
j
jxfxfxfxf
1
);();(ln..........
j
jnxfxf
Let T=u(X1, X2,….. Xn ) be an unbiased estimator of k() so
)(,....,)(21
kXXXuETEn
casecontinuoustheinisThat
......);().......;(),....,,(....)(21121 nnn
dxdxdxxfxfxxxuk
......);().......;(),....,,(....)(
21121 nnndxdxdxxfxfxxxuk
......);().......;(),....,,(....)(
21121 nnndxdxdxxfxfxxxuk
....);();(ln),..,,(...)(
21
11
21 n
n
j
j
n
j
jndxdxdxxfxfxxxuk
TZEkso )(
0);(...,1);(
xfgetwetrwderivativetakingdxxfNow
0);();(
);(
dxxfxf
xf
aswrittenbecanexpressionlatterThe
0);();(ln
dxxfxf
According to above we have
0);(ln);(ln
)(
11
nj
nj
xfE
xfEZE
,);(lnvar
1
n
j
jxfbyZiablerandomtheDefine
,variancesntheofsumtheisZofvarianceHence
2
);(ln)(
xfnEZVar
2
);(ln
xfEeh variancuently witand conseqmean zero
h each wit r.v. n indep. ofis the sum Z Moreover,
ZTZETETZEthatRecall )()()(
0)(),()(,)()( ZEkTEkTZESince
ZT
ZT
korkkhaveWe
)(0).()(
1
)()(
)(,11,1
2
2
ZVarTVar
kHencebecauseNow
2
22
;(ln
)()(..,)(
)(
)(
xfnE
kTVarhaveweeiTVar
ZVar
kor
.);();(ln
)(.)('
)(,);(ln
:
2
2
dxxfxf
IisThataboutninformatiosfisher
calledisIbydenotedxf
EexpressionTheDefinition
.
,
.);();(ln
)()(2
2
expressionsecondtheprefer
weoftenbutotherthethancomputetoeasierisexpressiononeSometimes
dxxfxf
IfromcomputedbecanI
)()(... nIIeinobservatioonein
ninformatiofisherthetimesnisnsizeofsamplerandomainninformatioFisher
n
)(
)()(
2
nI
kTVarbecomesInequalityRaoCramerSo
)(
1)(,1)(
)(
nITVarksince,becomesinequality
RaoCramerthethen,kthatso,θofestimatorunbisedanisTIf
Minimum Variance Bound Unbiased Estimator (MVBUE):
An unbiased estimator T of a parameter k() is called an MVBUE if Var(T)
attains Cramer-Rao lower bound.
.)(
)(
)()(,),()(ln
,)(
.
MVBUEbetokparameterofT
estimatoranforconditionsufficientandnecessarythethereforeisThis
A
kTVarkTAL
relationthesatisfymustitthenkparametertheofMVBUEisTIf
MVBUEtheforCriterion
)(
1)(
)()2
)(
)()(
)()()()1
:
2
nITVar
andTEifMVBUEbetosaidisofTestimatorAn
nI
kTVar
andkTEifMVBUEbetosaidiskofTestimatorAn
wordsotherIn
Result: If MVBUE exists for , then MVBUE exists for
k(), provided k is linear function.
:Pr oof
)(
1)()(,
nITVarandTEHenceMVBUEisTLet
bakandbaTTkLet )()(
)()(
)())((())((
22
nI
a
nI
kTkVarandbaTkEshowtohaveWe
)()()1 TESincebabaTE
)(
1)(
)(
1)()()2
22
nITVarsince
nIaTVarabaTVar
.)(kforexistMVBUEHence
Result : An MVBUE of a parameter k() must be a sufficient
statistic for k()
Proof:
)(
)()(,),()(ln
,)(
A
kTVarkTALrelation
thesatisfymustitthenkparametertheofMVBUEisTLet
getweCdAwritingandpartsbytrwitgIntegratin )()()(...
)()()()()()(ln xBdCkCkxTL
)()()()()(ln xBDCkxTL
)(
)(
),())(();(
kparameter
forstatisticsufficientaisXTthionfactorizat
NeymanbyfindwexhxtgxLwithitCompare
m
)()()()()(exp xBDCkxTL
)(exp)()()()(exp xBDCkxTL
.
,,),,(22
ofMVBUEtheismeansamplethe
thatshowknownisandwhereNFor
x
xxfSolution
2
2
2 2
)(exp
2
1);(:
2
22/1
2
2
)(2ln);(ln
xxf
2
2
2
2
)(2ln
2
1);(ln
xxf
222
)1)((2);(ln
xxxfThus
22
2
22
21
);(ln)(;1
);(ln
xfEIxfand
Example 1:
)(
1)(
)(
nITVar
andTEifMVBUEbetosaidisofTestimatorAn
),(~)(
2
nNXsinceXE
nXVar
2
)(
2
1
n
2
1)(
)(
1
Ibecause
nI
forMVBUEisXHence
.),,1(~ forMVBUEisXthatshowBXLet
1,0)1();(:1
xxfSolutionxx
)1ln()1(ln);(ln xxxf
1
1);(ln
xxxfThus
22222
2
)1(
)()1()(
)1(
)1();(ln)(
XEEXEXXE
xfEI
222
2
1
1);(ln
xxxfand
)1(
1
)1(
)1(
1
11
)1(
1)(..
22
Iei
Example 2:
)(
1)(
)(
nITVar
andTEifMVBUEbetosaidisofTestimatorAn
),(~1
)(
1
1 nBXsincen
nn
X
EXE
n
i
i
n
i
i
)1(
1)(
)(
1
Ibecause
nI
forMVBUEisXHence
)1(
1)1()1(
1)(
2
1
nnn
nn
X
VarXVar
n
i
i
.)(~ forMVBUEisXthatshowPXLet
,.........4,3,2,1,0!
);(:
xx
exfSolution
x
!lnln);(ln xxxf
1);(ln
xxfThus
1)();(ln)(
2222
2
XEXE
xfEI
22
2
);(ln
xxfand
Example 3:
)(
1)(
)(
nITVar
andTEifMVBUEbetosaidisofTestimatorAn
)(~1
)(
1
1 nPXsincen
nn
X
EXE
n
i
i
n
i
i
1)(
)(
1 Ibecause
nI
forMVBUEisXHence
nnn
nn
X
VarXVar
n
i
i11
)(2
1
.)(~ forMVBUEisXthatshowPXLet
Example 3: (Alternative approach)
.)(
)(
)()(,),()(ln
,)(
.
MVBUEbetokparameterofT
estimatoranforconditionsufficientandnecessarythethereforeisThis
A
kTVarkTAL
relationthesatisfymustitthenkparametertheofMVBUEisTIf
MVBUEtheforCriterion
,.........4,3,2,1,0!
);(
xx
exf
x
!lnln);(ln xxxf
);(...).........;();(21
n
xfxfxfL
);(............);(ln);(lnln21
n
xfxfxfL
);(lnln i
xfL
)!ln(ln)(ln)!ln(ln)(ln xxxfsincexxnLii
ix
nL
ln
xnnL
ln
x
nLln
)()(ln
kTALwithCompare
)(
1
)(
)()(.
AA
kTVarSinceofMVBUEanisXTthatfindWe
n
.,
),(2
knownbeingexistsondistributi
NofparametertheofMVBUEthewhetherDetermine
Example 4
.)(
)(
)()(,),()(ln
,)(
.
MVBUEbetokparameterofT
estimatoranforconditionsufficientandnecessarythethereforeisThis
A
kTVarkTAL
relationthesatisfymustitthenkparametertheofMVBUEisTIf
MVBUEtheforCriterion
2
)(exp
2
12
1
2
i
n
i
xL
Let
2
)(exp
2
12
i
n
xL
2
)(2ln
2ln
2
ixn
L
2
2)(
2
1
2ln
i
xn
L
)()(ln
kTALwithCompare
nAA
kTVarSince
ofMVBUEanisn
xTthatfindWe
i
2
2
2
)(
1
)(
)()(
.)(
2
2)(
2
1
2
i
xn
nn
n
xn i
2
2
)(
2
Definition 1 : Let T be an unbiased estimator of a parameter in
such a case of point estimation. The statistic T is called an efficient
estimator of if and only if the variance of T attains the Cramer-
Rao lower bound.
Definition 2: In cases in which we can differentiate with respect to
a parameter under an integral or summation symbol, the ratio of
the Cramer-Rao lower bound to the actual variance of any
unbiased estimator of a parameter is called the efficiency of that
statistic.
.1
.),,(
1
2
2
ofn
nSestimatortheof
efficiencytheisWhatknowniswhereNisthatondistributi
afromnsizeofsamplerandomaofvarincethedenoteSLet
1
)4(.2
:
2
2
n
nSEthatknowWe
exampleRefern
isboundlowerCramerSolution
)1(2var~
2
2
1
2
nisnS
ofiancethesonS
Nown
)1(2)1(
)1(,
2
n
n
nSnVaryAccordingl
1
2
)1()1(2
)1(
)1(..
222
2
2
nn
nSVarn
n
nSVar
nei
n
n
n
nisn
nSestimatortheofefficiencytheThus
1
1
2
2
12
2
2
![Index [assets.cambridge.org]assets.cambridge.org/97805215/14217/index/9780521514217...Index More informatio n 206 Index Gould, Lewis 1912 elections, 171 antitrust campaign of Taft,](https://static.documents.pub/doc/80x56/5e5e8ef0e9893b5ca2142e65/index-index-more-informatio-n-206-index-gould-lewis-1912-elections-171.jpg)
