Sufficient statistics.
The Poisson and the exponential can be summarized by (n, ).
So too can the normal with known variance
Consider a statistic S(Y)
Suppose that the conditional distribution of Y given S does not depend on , then S is a sufficient statistic for based on Y
Occurs iff the density of Y factors into a function of s(y) and and a function of y that doesn't depend on
y
More Chapter 4
Example. Exponential
IExp() ~ Y
E(Y) = Var(Y) = 2
Data y1,...,yn
L() = -1 exp(-yj /)
l() = -nlog() - yj /
yj /n is sufficient
2222
2
322
2
2
2)ˆ(
2)(
m.l.e. ˆ
1
0)ˆ( .
yn
yn
ynl
nnl
y
ynl
UequationLikelihood
maximum
0})(
2{)}({
)}({)(
.
2)(
.
2000
0
2
32
ynn
EUE
n
JEI
nInformatioFisher
ynn
J
nInformatioObserved
=
))(
,(~ˆ
distin )()(
)()(
probin 1)2
()()()(
)/()}({
)/()()}({
200
0
20
002/10
3
0
2
0
1
2
0
010
200
0
2000
0
nN
Zyn
nUI
ynnnn
JI
nJE
nIUVar
Approximate 100(1-2 )% CI for 0
2
2/1
/)ˆ()ˆ(
)ˆI(insert Could
)ˆ(ˆ
ynJIHere
Jz
Example. spring data
8.34)(168.26,16 )0188(.96.130.168
000353.3.168/10/)ˆ()ˆ(
3.168ˆ
22
ynJI
cyclesky
Weibull.
)/log()/(/log/
)/(/),(
log)1(loglog),(
)(exp),;()(
1 if lExponentia
0,, ,)(exp),;(
1
1
1
jjj
j
jj
j
jn
n
yyyn
ynU
yynnl
yyyfL
yyy
yf
Note.
),ˆ(),(max
problemmax D-1
)(ˆ /11
lll
yn
profile
j
Expected information
large Want
/)(log)(
/)(2)(2)'(1 (2)/-(2)/- )/(
),(22
2
I
dzzdz
nI
Gamma.
sufficient ),log(
log)1()(loglog)(
)exp()(
)(
0,, ),exp()(
),;(
1
1
jj
jj
jjn
n
yyS
yynnl
yyL
yyy
yf
Example. Bernoulli
Pr{Y = 1} = 1 - Pr{Y = 0} = 0 1
L() = ^yi (1 - )^(1-yi)
= r(1 - )n-r
l() = rlog() + (n-r)log(1-)
r = yj
R = Yj is sufficient for , as is R/n
L() factors into a function of r and a constant
Score vector
[ yj / - (n-yj )/(1-)]
Observed information
[yj /2 + (n-yj )/(1-)2 ]
ny j /ˆ
M.l.e.
Cauchy.
ICau()
f(y;) = 1/(1+(y-)2 )
E|Y| = Var(Y) =
L() = 1/((1+(yj -)2 )
Many local maxima
l() = -log(1+(yj -)2 )
J() = 2((1-(yj -)2 )/(1+(yj -)2 )2 I() = n/2
sufficient is ,....y
N(0,1) closer to is
)ˆ()ˆ( Z)ˆ()ˆ(
)((1)
0
2/1
J0
2/1
n
J
I
y
Z
JIZ
Uniform.
f(u;) = 1/ 0 < u <
= 0 otherwise
L() = 1/n 0 < y1 ,..., yn <
= 0 otherwise
0ˆ//)ˆ(
0/)ˆ(
,...y y
)max(ˆ
222
1(n)
nddl
ddl
y
y
n
j
l() becomes increasingly spikey
E u() = -1 i() = -
ondistributiin lExponentia)ˆ(
1
0 )/(
0 }ˆPr{
n
a
aa
aan
Logistic regression. Challenger data Ibinomials Rj , mj , j
)21()ˆ)(ˆ,ˆ()ˆ(
region Confidence
),(
statistic Sufficient
))exp(1(
)exp(
)!(!
!
),;Pr(),(
})exp{1/(}exp{
2001000
1
110
110
1010
110110
cJ
xRRS
x
xrr
rmr
m
rRL
xx
T
jjj
m
j
jjj
jjj
j
jj
jjj
j
Likelihood ratio.
Model includes dim() = p
true (unknown) value 0
Likelihood ratio statistic
)( ason distributiin )(
)}()ˆ({2)(
020
00
IW
llW
p
Justification.
Multinormal result
If Y ~ N (,) then (Y- )T -1(Y- ) ~ p2
)ˆ)(ˆ()ˆ(
)ˆ)(ˆ()ˆ(
)ˆ()ˆ(
)ˆ(21
)ˆ(
)ˆ()ˆ()ˆ({2
)}()ˆ({2)(
00
00
02
00
00
I
J
llll
llW
T
T
T
TT
Uses.
Pr[W(0) cp(1-2 )] 1-2
)}21(21
)ˆ()(:{
)21( )(
p
p
cll
cW
Approx 100(1-2 )% confidence region
Example. exponential
84.3}/log1/{2:{
84.3)95.0( 1
}/log1/{2)}()ˆ({2
log)ˆ(
/log)(
1
000
yyn
cp
yynll
nynl
ynnl
Spring data: 96 < <335
vs. asymp normal approx 64 < <273 kcycles
Prob-value/P-value. See (7.28)
Choose T whose large values cast doubt on H0
Pr0(T tobs)
Example. Spring data
Exponential E(Y) =
H0: = 100?
.071.0368*2
)802.1|Pr(|)248.3(Pr
248.3)100(
}/log1/{2)(
/)ˆ()ˆ(
10n 3.168ˆ
2
10
2
ZvalueP
W
yynW
ynJI
y
Nesting
: p by 1 parameter of interest
: q by 1 nuisance parameter
Model with params (0, ) nested within (, )
Second model reduces to first when = 0
)ˆ,()ˆ,ˆ(
Note.
0
0
ll
Example. Weibull
params (,)
exponential when = 1
How to examine H0 : = 1?
1p on,distributiin
)]ˆ,()ˆ,ˆ([2)(
2
p
000
llWp
Spring failure times. Weibull
07-5.73E
07)-2(2.867E
)00.5|(|
02.25)]1,168()6,181([2
26.61)1,168( 2749.2)1,168(
)1,168()1,( 75.48)6,181(
227.6)6,181( )6,181()ˆ,ˆ(
1
ZPvalueP
ll
lEL
l
EL
Challenger data. Logistic regression
temperature x1 pressure x2
(0 , 1 , 2 ) = exp{}/(1+exp{})
= 0 + 1 x1 + 2 x2 linear predictor
loglike l(0 , 1 , 2 ) =
0 rj + 1 rj x1j + 2 rj x2j - m log(1+exp{j })
Does pressure matter?
214.)107(.2)24.1|Pr(|
)54.1(Pr :
54.177.*2
05.15),,(max
82.15)0,,(max
2
10
210,,
10,
210
10
Z
valueP
l
l
Model fit.
Are labor times Weibull?
Nest its model in a more general one
Generalized gamma.
0,,, ),exp()(
),,;(1
yyy
yf
Gamma for =1
Weibull for =1
Exponential for ==1
Likelihood results.
max log likelihood:
generalized gamma -250.65
gamma -251.12
Weibull -251.17
gamma vs. generalized gamma
- 2 log like diff:
2(-250.65+251.12) = .94
P-value Pr0 (12 > .94)
= Pr(|Z|>.969)
= 2(.166) = .332
Chi-squared statistics. Pearson's chi-squared
categories 1,...,k
count of cases in category i: Yi
Pr(case in i) = i 0 < i < 1 1k i =1
E(Yi ) = ni
var(Yi ) = i (1 - i )n
cov(Yi ,Yj ) = -i j n i j
E.g. k=2 case cov(Y,n-Y) = -var(Y) = -n1 2
= { (1 ,...,k ): 1k i = 1, 0<1 ,...,k <1}
dimension k-1
Reduced dimension possible?
model i () dim() = p
log like general model:
1k-1 yi log i + yk log[1-1 -...-k-1], 1
k yi = n
nYii /ˆ
log like restricted model:
l() = 1k-1 yi log i() + yk log[1-1()-...-k-1()]
likelihood ratio statistic:
k
pkiiiy1
2
1~)ˆ(/ˆlog2
if restricted model true
The statistic is sometimes written
W = 2 Oi log(Oi /Ei )
(Oi - Ei )2/Ei
)ˆ(E where i iii nyO
Pearson's chi-squared.
5)ˆ(ntion recommenda
~
)ˆ(/)]ˆ([2
p-1-k
12
i
kiii nnyP
Example. Birth data. Poisson?
12.9ˆ 92n arrivalsDaily
Split into k=13 categories
[0,7.5), [7.5,8.5),...[18.5,24] hours
O(bserved) 6 3 3 8 ...
E(xpected) 5.23 4.37 6.26 8.08 ...
P = 4.39
P-value Pr(112 > 4.39) = .96
Two way contingency table.
r rows and c columns
n individuals
Blood groups A, B, AB, O
A, B antigens - substance causing body to produce antibodies
2
2
2
)1)(1(202
26
2)1(35
2)1(179
O
BA
OBB
OAA
O
AB
B
A
group count model I model II
O = 1 - A - B
Question. Rows and columns independent?
W = 2 yij log nyij / yi.y.j
with yi. = j yij
~ k-1-p2 = (r-1)c-1)
2
with k=rc p=(r-1)+(c-1)
P = (yij - yi. y.j /n)2 / (yi. y.j /n)
~ (r-1)(c-1)2
Model 1
W = 17.66
Pr(12> 17.66) = Pr(|Z| > 4.202) = 2.646E-05
P = 15.73
Pr(12> 15.73) = Pr(|Z| > 3.966) = 7.309E-05
k-1-p = 4-1-2 = 1
Model 2
W = 3.17
Pr(|Z| > 1.780) = .075
P = 2.82
Pr(|Z|>1.679) = .093
Incorrect model.
True model g(y), fit f(y;)
valuebad"least " :
0D 0);(
yprobabilitin )()f(y; log /)ˆ(
ydiscrepancLiebler -Kullback
)(})f(y;
g(y)log{ );( minimizes
);( log )( maximizes ˆ
g
g
g
j
ffD
dyygnl
dyyggfD
yfl
Example 1. Quadratic, fit linear
Example 2. True lognormal, but fit exponential
dyyg
nYY
y
ZY
gg
g
)(})f(y;
g(y)log{ Minimizing
/)1(var }2/exp{YE ˆ
}2/exp{
/log :likelog
}exp{ Lognormal
222
2
Large sample distribution.
);()( ifresult mle
))()()(;( ~ ˆ 11
yfyg
IKIN ggggp
Model selection.
Various models:
non-nested
Ockham's razor.
Prefer the simplest model
Formal criteria.
}log)ˆ({2
})ˆ({2
nplBIC
plAIC
Look for minimum
Example. Spring failure
Model p AIC BIC
M1 12 744.8* 769.9*
M2 7 771.8 786.5
M3 2 827.8 831.2
M4 2 925.1 929.3
6 stress levels
M1: Weibull - unconnected , at each stress level
