Date post: | 07-Jul-2018 |
Category: |
Documents |
Upload: | potado-tomado |
View: | 222 times |
Download: | 0 times |
of 246
8/18/2019 Differential Geometry in Statistical Inference
1/246
Institute of Mathematical Statistics
LECTURE NOTES-MONOGRAPH SERIES
Differential Geometry in
Statistical Inference
S.-l.
Am ari O. E. Barndorff-Nielsen
R. E. Kass S. L. Lauritzen and C. R. Rao
8/18/2019 Differential Geometry in Statistical Inference
2/246
8/18/2019 Differential Geometry in Statistical Inference
3/246
Institute of Mathematical Statistics
LECTURE NOTES-MONOGRAPH SERIES
Shanti S. Gupta Series Editor
Volume
10
Differential Geometry in
Statistical Inference
S.-l. Amari O. E. Barndorff-Nielsen
R. E. Kass S. L. Lauritzen and C. R. Rao
Institute of Mathem atical Statistics
Hayward California
8/18/2019 Differential Geometry in Statistical Inference
4/246
Institute of Mathematical Statistics
Lecture Notes-Monograph Series
Series Editor Shanti S. Gupta Purdue University
The production of the IMS Lecture Notes-Monograph Series is
managed by the IMS Business Office: Nicholas P. Jewell IMS
Treasurer and Jose L. Gonzalez IMS Business Manager.
Library o f C ongress Catalog Card Number: 87 -826 03
International Standard Book Number 0-94 060 0-12-9
Copyright © 1987 Institute of Mathematical Statistics
All r ights reserved
Printed in the United States of America
8/18/2019 Differential Geometry in Statistical Inference
5/246
TABLE OF CONTENTS
CHA PTER 1. Introduction
Robert E. Kass 1
CHAPTER 2. Differential Geometrical Theory of Statistics
Shun ichi Am ari 19
CH AP TER 3. Differential and Integral Geometry in Statistical Inference
O. E. Bamdorff Nielsen 95
CHAPTER 4. Statistical Manifolds
Steffen L. Lauritzen 163
CHAPTER 5. Differential Metrics in Probability Spaces
C. R. Rao 217
in
8/18/2019 Differential Geometry in Statistical Inference
6/246
8/18/2019 Differential Geometry in Statistical Inference
7/246
C H A P T E R 1 . I N T R O D U C T I O N
R o b e r t E . K a s s *
G e o m e t r i c a l a n a l y s e s o f p a r a m e t r i c i n f e r e n c e p r o b l e m s h a v e d e v e l o p e d
f r o m t w o a p p e a l i n g i d e a s : t h a t a l o c a l m e a s u r e o f d i s t a n c e b e t w e e n m e m b e r s o f a
f a m i l y o f d i s t r i b u t i o n s c o u l d b e b a s e d o n F i s h e r i n f o r m a t i o n , a n d t h a t t h e
s p e c i a l p l a c e o f e x p o n e n t i a l f a m i l i e s i n s t a t i s t i c a l t h e o r y c o u l d b e u n d e r s t o o d
a s b e i n g i n t i m a t e l y c o n n e c t e d w i t h t h e i r l o g l i n e a r s t r u c t u r e . T h e f i r s t l e d
J e f f r e y s ( 1 9 4 6 ) a n d R a o ( 1 9 4 5 ) t o i n t r o d u c e a R i e m a n n i a n m e t r i c d e f i n e d b y
F i s h e r i n f o r m a t i o n , w h i l e t h e s e c o n d l e d E f r o n ( 1 9 7 5 ) t o q u a n t i f y d e p a r t u r e s
f r o m e x p o n e n t i a l i t y b y d e f i n i n g t h e c u r v a t u r e o f a s t a t i s t i c a l m o d e l . T h e
p a p e r s c o l l e c t e d i n t h i s v o l u m e s u m m a r i z e s u b s e q u e n t r e s e a r c h c a r r i e d o u t b y
P r o f e s s o r s A m a r i , B a r n d o r f f - N i e l s e n , L a u r i t z e n , a n d R a o t o g e t h e r w i t h t h e i r
c o w o r k e r s , a n d b y o t h e r a u t h o r s a s w e l l , w h i c h h a s s u b s t a n t i a l l y e x t e n d e d b o t h
t h e a p p l i c a b i l i t y o f d i f f e r e n t i a l g e o m e t r y a n d o u r u n d e r s t a n d i n g o f t h e r o l e i t
p l a y s i n s t a t i s t i c a l t h e o r y . * *
T h e m o s t b a s i c s u c c e s s o f t h e g e o m e t r i c a l m e t h o d r e m a i n s i t s c o n c i s e
s u m m a r y o f i n f o r m a t i o n l o s s , F i s h e r ' s f u n d a m e n t a l q u a n t i f i c a t i o n o f d e p a r t u r e
f r o m s u f f i c i e n c y , a n d i n f o r m a t i o n r e c o v e r y , h i s j u s t i f i c a t i o n f o r c o n d i t i o n i n g .
F i s h e r c l a i m e d , b u t
r \ e \ e r
s h o w e d , t h a t t h e NILE m i n i m i z e d t h e l o s s o f i n f o r m a t i o n
a m o n g e f f i c i e n t e s t i m a t o r s , a n d t h a t s u c c e s s i v e p o r t i o n s o f t h e l o s s c o u l d b e
*
D e p a r t m e n t o f S t a t i s t i c s , C a r n e g i e - M e l l o n U n i v e r s i t y , P i t t s b u r g h , P A
**
T h e s e p a p e r s w e r e p r e s e n t e d a t t h e N A T O A d v a n c e d W o r k s h o p o n D i f f e r e n t i a l
G e o m e t r y i n S t a t i s t i c a l I n f e r e n c e a t I m p e r i a l C o l l e g e , A p r i l , 1 9 3 4 .
8/18/2019 Differential Geometry in Statistical Inference
8/246
2 Robert E. Kass
r e c o v e r e d by c o n d i t i o n i n g on t h e s e c o n d a n d h i g h e r d e r i v a t i v e s of t h e l o g
l i k e l i h o o d f u n c t i o n , e v a l u a t e d
at
t h e M L E . C o n c e r n i n g i n f o r m a t i o n l o s s , r ec a l l
t h a t a c c o r d i n g
to
t h e K o o p m a n D a r m o i s t h e o r e m , u n d e r r e g u l a r i t y c o n d i t i o n s ,
the
f a m i l i e s
of
c o n t i n u o u s d i s t r i b u t i o n s w i t h f i x e d s u p p o r t t h a t a d m i t f i n i t e
d i m e n s i o n a l s u f f i c i e n t r e d u c t i o n s
of
i . i . d . s e q u e n c e s a r e p r e c i s e l y t h e e x p o n e n
t i a l f a m i l i e s . It is t h u s i n t u i t i v e t h a t ( f or s u c h r e g u l a r f a m i l i e s ) d e p a r t u r e s
f r o m s u f f i c i e n c y , t h a t i s , i n f o r m a t i o n l o s s , s h o u l d c o r r e s p o n d to d e v i a t i o n s
f r o m e x p o n e n t i a l i t y . T h e r e m a r k a b l e r e a l i t y
is
t h a t t h e c o r r e s p o n d e n c e t a k e s
a
b e a u t i f u l l y s i m p l e f o r m . T h e m o s t t r a n s p a r e n t c a s e , e s p e c i a l l y f o r t h e u n t r a i n
e d e y e , o c c u r s f o r
a
o n e p a r a m e t e r s u b f a m i l y
of a
t w o d i m e n s i o n a l e x p o n e n t i a l
f a m i l y . T h e r e , t h e r e l a t i v e i n f o r m a t i o n l o s s , in F i s h e r ' s s e n s e , f r o m u s i n g a
s t a t i s t i c T in p l a c e of t h e w h o l e s a m p l e is
1 1 m i ( θ )
Ί
[ n i ( θ ) i
T
( θ ) ]
= γ
2
+
\
3
2
(1)
w h e r e n i ( θ ) is t h e F i s h e r i n f o r m a t i o n in t h e w h o l e s a m p l e , i ( θ ) is t h e F i s h e r
i n f o r m a t i o n c a l c u l a t e d f r o m t h e d i s t r i b u t i o n
of
T ,
γ is
t h e s t a t i s t i c a l c u r v a
t u r e
of
t h e f a m i l y a n d
3 is
t h e m i x t u r e c u r v a t u r e
of
t h e a n c i l l a r y f a m i l y
a s s o c i a t e d w i t h t h e e s t i m a t o r T . W h e n t h e e s t i m a t o r
T is
t h e M L E ,
3
v a n i s h e s ;
t h i s s u b s t a n t i a t e s F i s h e r ' s f i r s t c l a i m .
In h i s 1 9 7 5 p a p e r , E f r o n d e r i v e d t h e t w o t e r m e x p r e s s i o n f o r i n f o r
m a t i o n l o s s ( i n h i s e q u a t i o n ( 1 0 . 2 5 ) ) , d i s c u s s e d t h e g e o m e t r i c a l i n t e r p r e t a t i o n
o f t h e f i r s t t e r m , a n d n o t e d t h a t t h e s e c o n d t e r m
is
z e r o f o r t h e M L E .
He
d e f i n e d
γ to be
t h e c u r v a t u r e
of
t h e c u r v e
in
t h e n a t u r a l p a r a m e t e r s p a c e t h a t
d e s c r i b e s t h e s u b f a m i l y , w i t h t h e i n n e r p r o d u c t d e f i n e d
by
F i s h e r i n f o r m a t i o n
r e p l a c i n g t h e u s ua l E u c l i d e a n i n n e r p r o d u c t . T h e d e f i n i t i o n
of 3 is
e x a c t l y
a n a l o g o u s to t h a t of γ , w i t h t h e m e a n v a l u e p a r a m e t e r s p a c e u s e d i n s t e a d of t h e
n a t u r a l p a r a m e t e r s p a c e , b u t E f r o n d i d n o t r e c o g n i z e t h i s a n d
so
d i d
ot
i d e n t i f y t h e m i x t u r e c u r v a t u r e .
He
d i d s t r e s s t h e r o l e
of
t h e a n c i l l a r y f a m i l y
a s s o c i a t e d w i t h t h e e s t i m a t o r
T
( s e e h i s R e m a r k
3 of
S e c t i o n
9
a n d h i s r e p l y
to
d i s c u s s a n t s ,
p. 1 2 4 0 ) ,
a n d
he
a l s o n o t i c e d
a
s p e c i a l c a s e
of
( 1 ) ( in h i s r e p l y ,
p .
1 2 4 1 ) .
T h e f i n a l s i m p l i c i t y of t h e c o m p l e t e g e o m e t r i c a l v e r s i o n of (1)
8/18/2019 Differential Geometry in Statistical Inference
9/246
Introduction 3
a p p e a r e d in A m a r i ' s 1 9 8 2 A n n a l s p a p e r . T h e r e it w a s d e r i v e d in t h e m u l t i
p a r a m e t e r c a s e ; s e e e q u a t i o n ( 4 . 8 ) of A m a r i ' s p a p e r in t h i s v o l u m e .
P r i o r
to
E f r o n ' s p a p e r ,
Rao
1 9 6 1 )
had
i n t r o d u c e d d e f i n i t i o n s
of
e f f i c i e n c y a n d s e c o n d o r d e r e f f i c i e n c y t h a t w e r e i n t e n d e d to c l a s s i f y e s t i m a t o r s
j u s t
as F i s h e r ' s d e f i n i t i o n s d i d , b u t u s i n g m o r e t r a c t a b l e e x p r e s s i o n s . T h i s
l e d to t h e s a m e m e a s u r e of m i n i m u m i n f o r m a t i o n l o s s u s e d by F i s h e r ( c o r r e s p o n d
2
i n g to γ in e q u a t i o n ( 1 ) ) . Rao ( 1 9 6 2 ) c o m p u t e d t h e i n f o r m a t i o n l o s s in t h e
c a s e of t h e m u l t i n o m i a l d i s t r i b u t i o n f o r s e v e r a l d i f f e r e n t m e t h o d s of e s t i m a t i o n .
R a o ( 1 9 6 3 ) t h en
w e n t
on to p r o v i d e a d e c i s i o n t h e o r e t i c d e f i n i t i o n of s e c o n d
o r d e r e f f i c i e n c y
of an
e s t i m a t o r T , m e a s u r i n g
it
a c c o r d i n g
to
t h e m a g n i t u d e
of
t h e s e c o n d o r d e r t e r m in t h e a s y m p t o t i c e x p a n s i o n of t h e b i a s c o r r e c t e d v e r s i o n
o f T . E f r o n ' s a n a l y s i s c l a r i f i e d t h e r e l a t i o n s h i p b e t w e e n F i s h e r ' s d e f i n i t i o n
a n d R a o ' s
f i r s t
d e f i n i t i o n . E f r o n t h e n p r o v i d e d a d e c o m p o s i t i o n of t h e s e c o n d
o r d e r v a r i a n c e t e r m in w h i c h t h e r i g h t h a n d s i d e of ( 1 ) a p p e a r e d , t o g e t h e r w i t h
a p a r a m e t e r i z a t i o n d e p e n d e n t t h i r d t e r m . T h e e x t e n s i o n to t h e m u l t i p a r a m e t e r
c a s e w a s d e r i v e d by M a d s e n ( 1 9 7 9 ) f o l l o w i n g t h e o u t l i n e of R e e d s ( 1 9 7 5 ) . It
a p p e a r s h e r e in A m a r i ' s p a p e r as T h e o r e m 3 . 4 .
A n a n a l y t i c a l l y a n d c o n c e p t u a l l y i m p o r t a n t f i r s t s t e p of E f r o n ' s
a n a l y s i s w a s to b e g i n by c o n s i d e r i n g s m o o t h s u b f a m i l i e s of r e g u l a r e x p o n e n t i a l
f a m i l i e s , w h i c h he c a l l e d c u r v e d e x p o n e n t i a l f a m i l i e s . A n a l y t i c a l l y , t h i s m a d e
p o s s i b l e r i g o r o u s d e r i v a t i o n s of r e s u l t s , a n d f o r t h i s r e a s o n s u c h f a m i l i e s
w e r e a n a l y z e d c o n c u r r e n t l y by G h o s h a n d S u b r a m a n i a m
( 1 9 7 4 ) .
C o n c e p t u a l l y , it
a l l o w e d s p e c i f i c a t i o n
of
t h e a n c i l l a r y f a m i l i e s a s s o c i a t e d w i t h
an
e s t i m a t o r :
t h e a n c i l l a r y f a m i l y a s s o c i a t e d w i t h T at t is t h e s e t of p o i n t s y in t h e s a m p l e
s p a c e of t h e f u l l e x p o n e n t i a l f a m i l y e q u i v a l e n t l y , t h e m e a n v a l u e p a r a m e t e r
s p a c e f o r t h e f a m i l y f o r w h i c h T ( y ) = t. T h e t e r m i n o l o g y a nd s u b s e q u e n t
d e t a i l e d a n a l y s i s is d u e to A m a r i b u t , as n o t e d a b o v e , t h e i m p o r t a n c e of t h e
a n c i l l a r y f a m i l y , at o n c e e m p h a s i z e d a n d o b s c u r e d by F i s h e r , w a s a p p a r e n t f r o m
E f r o n ' s p r e s e n t a t i o n .
T h e a n c i l l a r y f a m i l y is a l s o i m p o r t a n t in u n d e r s t a n d i n g i n f o r m a t i o n
8/18/2019 Differential Geometry in Statistical Inference
10/246
4 Robert E. Kass
r e c o v e r y , w h i c h is t h e r e a s o n A m a r i h a s c h o s e n to u s e t h e m o d i f i e r a n c i l l a r y .
In t h e d i s c u s s i o n
of
E f r o n ' s p a p e r , P i e r c e ( 1 9 7 5 ) n o te d a n o t h e r i n t e r p r e t a t i o n
o f s t a t i s t i c a l c u r v a t u r e :
it
f u r n i s h e s t h e a s y m p t o t i c s t a n d a r d d e v i a t i o n
of
o b s e r v e d i n f o r m a t i o n . M o r e p r e c i s e l y ,
it is
t h e a s y m p t o t i c s t a n d a r d d e v i a t i o n
1 / 2
Λ
1
o f t h e a s y m p t o t i c a l l y a n c i l l a r y s t a t i s t i c
n
i ( θ ) [ I ( θ )
n i ( θ ) ] ,
w h e r e
n i ( θ )
is
e x p e c t e d i n f o r m a t i o n a n d I ( θ )
is
o b s e r v e d i n f o r m a t i o n ; t h e o n e
p a r a m e t e r s t a t e m e n t a p p e a r s in E f r o n a n d H i n k l e y ,
( 1 9 7 8 ) ,
a nd t h e m u l t i p a r a m e t e r
v e r s i o n is in S k o v g a a r d ( 1 9 8 5 ) . W h e n f i t t i n g a c u r v e d e x p o n e n t i a l f a m i l y by t h e
m e t h o d of m a x i m u m l i k e l i h o o d , t h i s s t a t i s t i c b e c o m e s a n o r m a l i z e d c o m p o n e n t of
t h e r e s id u a l ( in t h e d i r e c t i o n n o r ma l
to
t h e m o d e l w i t h i n t h e p l a n e s p a n n e d
by
t h e f i r s t t w o d e r i v a t i v e s
of
t h e n a t u r a l p a r a m e t e r f o r t h e f u l l e x p o n e n t i a l
f a m i l y ) . F u r t h e r m o r e , c o n d i t i o n i n g on t h i s s t a t i s t i c r e c o v e r s ( in F i s h e r ' s
s e n s e ) t h e i n f o r m a t i o n l o s t by t h e N I LE , at l e a s t a p p r o x i m a t e l y . W h e n t h i s
c o n d i t i o n a l d i s t r i b u t i o n is u s e d , t h e a s y m p t o t i c v a r i a n c e of t h e NILE m a y be
e s t i m a t e d by t h e i n v e r s e of o b s e r v e d r a t h e r t h a n e x p e c t e d i n f o r m a t i o n ; in s o m e
p r o b l e m s o b s e r v e d i n f o r m a t i o n
is
c l e a r l y s u p e r i o r .
T h i s a r g u m e n t , s k e t c h e d
by
P i e r c e a n d p r e s e n t e d
in
m o r e d e t a i l
by
E f r o n a n d H i n k l e y , r e p r e s e n t e d an a t t e m p t to m a k e s e n s e of s o m e of F i s h e r ' s
r e m a r k s on c o n d i t i o n i n g . In S e c t i o n 4 of h i s p a p e r in t h i s v o l u m e , A m a r i
p r e s e n t s a c o m p r e h e n s i v e a p p r o a c h to i n f o r m a t i o n r e c o v e r y as m e a s u r e d by F i s h e r
i n f o r m a t i o n .
He
b e g i n s
by
d e f i n i n g
a
s t a t i s t i c
T to be
a s y m p t o t i c a l l y s u f f i
c i e n t
of
o r d e r
q
w h e n
n i ( θ )
i
T
( θ )
=
0 ( n '
q + 1
)
a n d a s y m p t o t i c a l l y a n c i l l a r y of o r d e r q w h e n
i
T
( θ ) = 0 ( n
q
) .
T h e s e d e f i n i t i o n s d i f f e r f r o m s o m e u s e d by o t h e r a u t h o r s , s u c h as C o x ( 1 9 8 0 ) ,
M c C u l l a g h ( 1 9 8 4 a ) , a n d S k o v g a a r d
( 1 9 8 5 ) .
T h e y a r e , h o w e v e r , c l e a r l y
in
t h e
s p i r i t of F i s h e r ' s a p p a r e n t f e e l i n g t h a t i ( θ ) is an a p p r o p r i a t e m e a s u r e of
i n f o r m a t i o n .
To
a n a l y z e F i s h e r ' s s u g g e s t i o n
t h a t
h i g h e r d e r i v a t i v e s
of
t h e
l o g l i k e l i h o o d f u n c t i o n c o u ld
be
u s e d
to
c r e a t e s u c c e s s i v e h i g h e r o r d e r
8/18/2019 Differential Geometry in Statistical Inference
11/246
Introduction
5
a p p r o x i m a t e a n c i l l a r y s t a t i s t i c s , A m a ri d e f i n e s
an
e x p l i c i t s e q u e n c e
of
c o m b i n a t i o n s of t h e d e r i v a t i v e s : he t a k e s s u c c e s s i v e c o m p o n e n t s of t h e r e s i d u a l
i n s p a c e s s p a n n e d by t h e f i r s t p d e r i v a t i v e s of t h e n a t u r a l p a r a m e t e r f o r t h e
a m b i e n t e x p o n e n t i a l f a m i l y b u t p e r p e n d i c u l a r to t h e s p a c e s p a n n e d by t h e f i r s t
p 1 , t h e n n o r m a l i z e s
by
h i g h e r o r d e r c u r v a t u r e s .
In
T h e o r e m s 4 .1 a n d
4.2
A m a r i a c h i e v e s
a
c o m p l e t e d e c o m p o s i t i o n
of
t h e i n f o r m a t i o n .
He
t h e r e b y m a k e s
s p e c i f i c , j u s t i f i e s , a n d p r o v i d e s a g e o m e t r i c a l i n t e r p r e t a t i o n f o r F i s h e r ' s
s e c o n d c l a i m . In A m a r i ' s d e c o m p o s i t i o n t h e p t h t e r m is a t t r i b u t a b l e to t h e .
p t h s t a t i s t i c in h i s s e q u e n c e a n d h a s m a g n i t u d e e q u a l to n ^ t i m e s the
s q u a r e
of
t h e p t h o r d e r c u r v a t u r e . ( A c t u a l l y , A m a r i ' s t r e a t m e n t
is
m o r e
g e n e r a l t h a n t h e r o u g h d e s c r i p t i o n h e r e w o u l d i m p l y s i n c e
he
a l l o w s f o r t h e u s e
o f e f f i c i e n t e s t i m a t o r s o t h e r t h a n t h e M L E . )
A s f a r
as
t h e b a s i c i s s u e
of
o b s e r v e d v e r s u s e x p e c t e d i n f o r m a t i o n
is
c o n c e r n e d , A m ar i ( 1 9 8 2 b ) u s e d an E d g e w o r t h e x p a n s i o n i n v o l vi n g g e o m e t r i c a l l y
i n t e r p r e t a b l e t e r m s ( as in A m a r i and K u m o n , 1 9 8 3 ) to p r o v i d e a g e n e r a l m o t i v a
t i o n f o r u s i n g t h e i n v e r s e of o b s e r v e d i n f o r m a t i o n as t h e e s t i m a t e of t h e
c o n d i t i o n a l v a r i a n c e
of
t h e M L E . S e e S e c t i o n 4 . 4
of
t h e p a p e r h e r e . ( In t r u t h ,
t h e r e s u l t
is
n o t
as
s t r o n g
as it
m a y a p p e a r . W h e n
we
h a v e
an
a p p r o x i m a t i o n
v
t o a v a r i a n c e v s a t i s f y i n g v ( θ ) = v
n
( θ ) { l + 0 ( n ) } , a n d we u s e it to e s t i m a t e
v ( θ ) ,
we s u b s t i t u t e v ( θ ) , w h e r e θ is s o m e e s t i m a t o r of θ , a n d t h e n a l l we
Λ
I O
u s u a l l y g e t
is
v ( θ )
= v
( θ ) { Ί
+ 0 n )}.
F o r e s s e n t i a l l y t h i s r e a s o n ,
o b s e r v e d i n f o r m a t i o n d o e s n o t in g e n e r al p r o v i d e an a p p r o x i m a t i o n to t h e c o n
d i t i o n a l v a r i a n c e of t h e M L E b a s e d on t h e u n d e r l y i n g t r u e v a l u e θ, h a v i n g
r e l a t i v e e r r o r 0 n ) a l t h o u g h it d o e s do so w h e n e v e r e x p e c t e d i n f o r m a t i o n is
c o n s t a n t , as it is f o r a l o c a t i o n p a r a m e t e r . S i m i l a r l y , as S k o v g a a r d , 1 9 8 5 ,
p o i n t s o u t
in
h i s c a r e f u l c o n s i d e r a t i o n
of
t h e r o l e
of
o b s e r v e d i n f o r m a t i o n
in
i n f e r e n c e , w h e n e s t i m a t e d c u m u l a n t s a r e u s e d
in an
E d g e w o r t h e x p a n s i o n
it
l o s e s
i t s h i g h e r o r d e r a p p r o x i m a t i o n to t h e u n d e r l y i n g d e n s i t y at t h e t r u e v a l u e .
T h i s p r a c t i c a l l i m i t a t i o n
of
a s y m p t o t i c s d o e s n o t a f f e c t B a y e s i a n i n f e r e n c e ,
in
w h i c h o b s e r v e d i n f o r m a t i o n f u r n i s h e s a b e t t e r a p p r o x i m a t i o n to t h e p o s t e r i o r
8/18/2019 Differential Geometry in Statistical Inference
12/246
8/18/2019 Differential Geometry in Statistical Inference
13/246
Introduction 7
t o E f r o n ' s p a p e r , D a w i d ( 1 9 7 5 ) p o i n t e d o u t t h a t Ef r o n h a d u s e d t h e R i e m a n n i a n
m e t r i c d e f i n e d
by
F i s h e r i n f o r m a t i o n , b u t t h a t
he
h a d e f f e c t i v e l y u s e d
a
n o n
R ie ma rπ . a n a f f i n e c o n n e c t i o n , n o w c a l l e d t h e e x p o n e n t i a l c o n n e c t i o n , in c a l
c u l a t i n g s t a t i s t i c a l c u r v a t u r e . A l t h o u g h D a wi d d i d n o t i d e n t i f y t h e r o l e of t h e
m i x t u r e c u r v a t u r e in ( 1 ) , he d i d d r a w a t t e n t i o n to t h e m i x t u r e c o n n e c t i o n as an
a l t e r n a t i v e
to
t h e e x p o n e n t i a l c o n n e c t i o n . ( G e o d e s i e s w i t h r e s p e c t
to
t h e
e x p o n e n t i a l c o n n e c t i o n f o r m e x p o n e n t i a l f a m i l i e s , w h i l e g e o d e s i e s w i t h r e s p e c t
t o t h e m i x t u r e c o n n e c t i o n f o r m f a m i l i e s of m i x t u r e s ; t h u s , t h e t e r m i n o l o g y . )
A m a r i , w h o h a d m u c h e a r l i e r r e s e a r c h e d t h e R i e m a n n i a n g e o m e t r y of F i s h e r i n f o r
m a t i o n , p i c k e d
up on
D a w i d
1
s o b s e r v a t i o n , s p e c i f i e d t h e f r a m e w o r k , a n d p r o v i d e d
t h e r e s u l t s o u t l i n e d a b o v e .
T h e m a n i f o l d w i t h t h e a s s o c i a t e d l i n e a r s p a c e s is s t r u c t u r e d in w h a t
i s u s u a l l y c a l l e d
a
t a n g e n t b u n d l e , t h e e l e m e n t s
of
t h e l i n e a r s p a c e s b e i n g
t a n g e n t v e c t o r s . F o r c u r v e d e x p o n e n t i a l f a m i l i e s , t h e l i n e a r s p a c e s a r e f i n i t e
d i m e n s i o n a l , b u t to a n a l y z e g e n e r a l f a m i l i e s t h i s d o e s n o t s u f f i c e so A m a r i
u s e s H u b e r t s p a c e s . W h e n t h e s e a r e a p p r o p r i a t e l y g l u e d t o g e t h e r , t h e r e s u l t
i s a H u b e r t b u n d l e . T h e id e a s t e m s f r o m D a w i d
1
s r e m a r k t h a t t h e t a n g e n t
v e c t o r s c a n
be
i d e n t i f i e d w i t h s c o r e f u n c t i o n s , a n d t h e s e
in
t u r n a r e f u n c t i o n s
h a v i n g z e r o e x p e c t a t i o n .
As
h i s H u b e r t s p a c e
at a
d i s t r i b u t i o n
P,
A m a r i t a k e s
t h e s u b s p a c e of t h e u s u a l L p ( P ) H u b e r t s p a c e c o n s i s t i n g of f u n c t i o n s t h a t h a v e
z e r o e x p e c t a t i o n w i t h r e s p e c t to P. T h i s c l e a r l y f u r n i s h e s t h e e x t e n s i o n of
t h e i n f o r m a t i o n m e t r i c , a nd h a s b e e n u s ed by o t h e r a u t h o r s as w e l l , e . g . ,
B e r a n
( 1 9 7 7 ) .
A m a r i t he n d e f i n e s t h e e x p o n e n t i a l a n d m i x t u r e c o n n e c t i o n s
nd
n o t e s t h a t t h e s e m a k e t h e H u b e r t b u n d l e f l a t , a n d t h a t t h e i n h e r i t e d c o n n e c
t i o n s on t h e u s u al t a n g e n t b u n d l e s a g r e e w i t h t h o s e a l r e a d y d e f i n e d t h e r e . He
t h e n d e c o m p o s e s e a c h H u b e r t s p a c e i nt o t a n g e n t i a l a n d n o r m a l c o m p o n e n t s ,
w h i c h is e x a c t l y w h a t is n e e d e d to d e f i n e s t a t i s t i c a l c u r v a t u r e in t h e g e n e r a l
s e t t i n g . A m a r i g o e s
on to
c o n s t r u c t
an
e x p on e n t i a l b u n d l e
by
a s s o c i a t i n g
w i t h e a c h d i s t r i b u t i o n
a
f i n i t e d i m e n s i o n a l l i n e a r s p a c e c o n t a i n i n g v e c t o r s
d e f i n e d by h i g h e r d e r i v a t i v e s of t h e l o g l i k e l i h o o d f u n c t i o n , a n d u s i n g s t r u c t u r e
8/18/2019 Differential Geometry in Statistical Inference
14/246
8 Robert E. Kass
i n h e r i t e d f r o m t h e H u b e r t b u n d l e . W i t h t h i s
he
o b t a i n s
a
s a t i s f a c t o r y v e r s i o n
o f t h e l o ca l a p p r o x i m a t i o n by a c u r v e d e x p o n e n t i a l f a m i l y t h a t E f r o n h a d
s u g g e s t e d .
T h i s p r e t t y c o n s t r u c t i o n a l l o w s r e s u l t s d e r i v e d f o r c u r v e d e x p o n e n
t i a l f a m i l i e s to be e x t e n d e d to m o r e g e n e r a l r e g u l a r f a m i l i e s , y e t it is not
q u i t e t h e a l l e n c o m p a s s i n g s t r u c t u r e o n e m i g h t h o p e f o r : t h e u n d e r l y i n g
m a n i f o l d
is
s t i l l
a
p a r t i c u l a r p a r a m e t r i c f a m i l y
of
d e n s i t i e s r a t h e r t h a n t h e
c o l l e c t i o n
of
a ll p o s s i b l e d e n s i t i e s
on
t h e g i v e n s a m p l e s p a c e . C o n s t r u c t i o n s
f o r t h e l a t t e r h a v e so f a r p r o v e d t o o d i f f i c u l t .
In h i s A n n a l s p a p e r , A m a r i a l s o n o t e d
an
i n t e r e s t i n g r e l a t i o n s h i p
b e t w e e n t h e e x p o n e n t i a l a n d m i x t u r e c o n n e c t i o n s : t h e y a r e , in a s e n s e he
d e f i n e d , m u t u a l l y d u a l . F u r t h e r m o r e ,
a
o n e p a r a m e t e r f a m i l y
of
c o n n e c t i o n s ,
w h i c h A m a r i c a l l e d t h e α c o n n e c t i o n s , m a y
be
d e f i n e d
in
s u c h
a
w a y t h a t f o r e a c h
α t h e α c o n n e c t i o n a n d t h e α c o n n e c t i o n a r e m u t u a l l y d u a l , w h i l e α = l a n d 1
c o r r e s p o n d to t h e e x p o n e n t i a l a n d m i x t u r e c o n n e c t i o n s . S e e A m a r i ' s T h e o r e m 2 . 1 .
T h i s f a m i l y c o i n c i d e s w i t h t h a t i n t r o d u c e d by C e n t s o v ( 1 9 7 1 ) f o r m u l t i n o m i a l
d i s t r i b u t i o n s . W h e n t h e f a m i l y
of
d e n s i t i e s
on
w h i c h t h e s e c o n n e c t i o n s a r e
d e f i n e d
is an
e x p o n e n t i a l f a m i l y , t h e s p a c e
is
f l a t w i t h r e s p e c t
to
t h e e x p o n e n
t i a l a n d m i x t u r e c o n n e c t i o n s , a n d t h e n at u r a l p a r a m e t r i z a t i o n a n d m e a n v a l u e
p a r a m e t e r i z a t i o n p l a y s p e c i a l r o l e s : t h e y b e c o m e a f f i n e c o o r d i n a t e s y s t e m s f o r
t h e t w o r e s p e c t i v e c o n n e c t i o n s a n d a r e r e l a t e d by a L e g e n d r e t r a n s f o r m a t i o n .
T h e d u a l i t y in t h i s c a s e c a n i n c o r p o r a t e t h e c o n v e x d u a l i t y t h e o r y of e x p o n e n
t ia l f a m i l i e s ( s e e B a r n d o r f f N i e l s e n , 1 9 7 8 , a nd a l s o S e c t i o n
2 of
h i s p a p e r
in
t h i s
v o l u m e ) . In
T h e o r e m 2 . 2 A m a r i p o i n t s o u t t h a t s u c h
a
p a i r
of
c o o r d i n a t e
s y s t e m s e x i s t s w h e n e v e r a s p a c e is f l a t w i t h r e s p e c t to an α c o n n e c t i o n ( w i t h
α
f
0 ) . F o r s u c h s p a c e s , A m a ri d e f i n e s α d i v e r g e n c e , a q u a s i d i s t a n c e b e t w e e n
t w o m e m b e r s of t h e f a m i l y b a s e d on t h e r e l a t i o n s h i p p r o v i d e d by t h e L e g e n d r e
t r a n s f o r m a t i o n .
In
T h e o r e m 2 . 4
he
s h o w s t h a t t h e e l e m e n t
of a
c u r v e d e x p o n e n t ia l
f a m i l y t h a t m i n i m i z e s t h e α d i v e r g e n c e f r o m
a
p o i n t
in
t h e e x p o n e n t i a l f a m i l y
p a r a m e t e r s p a c e m a y be f o u n d by f o l l o w i n g t h e α g e o d e s i c t h a t c o n t a i n s t h e
8/18/2019 Differential Geometry in Statistical Inference
15/246
Introduction 9
g i v e n p o i n t a n d is p e r p e n d i c u l a r to t h e c u r v e d f a m i l y . T h i s g e n e r a t e s a n e w
c l a s s of m i n i m u m α d i v e r g e n c e e s t i m a t o r s , t h e M L E b e i ng t h e m i n i m u m
1 d i v e r g e n c e e s t i m a t o r ,
an
i n t e r p r e t a t i o n a l s o d i s c u s s e d
by
E f r o n ( 1 9 7 8 ) .
A s a p p l i c a t i o n s
of
h i s g e n e r a l m e t h o d s b a s e d
on
α c o n n e c t i o n s
on
H u b e r t b u n d l e s , A m a r i t r e a t s t h e p r o b l e m s of c o m b i n i n g i n d e p e n d e n t s a m p l e s at
t h e e n d of s e c t i o n 5 ) , m a k i n g i n f e r e n c e s w h e n t h e n u m b e r of n u i s a n c e p a r a m e t e r s
i n c r e a s e s w i t h t h e s a m p l e s i z e ( in s e c t i o n 6 ) , a n d p e r f o r m i n g s p e ct r a l e s t i m a
t i o n
in
G a u s s i a n t i m e s e r i e s ( in s e c t i o n 7 ) .
A s s o o n
as
t h e α c o n n e c t i o n s a r e c o n s t r u c t e d
a
m a t h e m a t i c a l q u e s t i o n
a r i s e s .
On
o n e h a n d , t h e α c o n n e c t i o n s m a y
be
c o n s i d e r e d o b j e c t s
of
d i f f e r e n
t ia l g e o m e t r y w i t h o u t s p e c i a l r e f e r e n c e to t h e i r s t a t i s t i c a l o r i g i n . On t h e
o t h e r h a n d , t h e y a r e n o t at a ll a r b i t r a r y . T h e y a r e t h e s i m p l e s t o n e p a r a m e t e r
f a m i l y of c o n n e c t i o n s b a s e d on t h e f i r s t t h r e e m o m e n t s of t h e s c o r e f u n c t i o n .
W h a t
is it
a b o u t t h e i r s p e c i a l f o r m t h a t l e a d s
to
t h e m a n y s p e c i a l p r o p e r t i e s
o f α c o n n e c t i o n s ( o u t l i n e d
by
A m a r i
in
S e c t i o n
2 1
L a u r i t z e n h a s p o se d t h i s q u e s t i o n a n d h a s p r o v i d e d
a
s u b s t a n t i a l
p a r t of t h e a n s w e r . G i v e n a n y R i e m a n n i a n m a n i f o l d M w i t h m e t r i c g t h e r e is a
u n i q u e R i e m a n n i a n c o n n e c t i o n v. G i v e n a c o v a r i a n t 3 t e n s o r D t h a t is s y m m e t r i c
i n i ts f i r s t t w o a r g u m e n t s a n d a n o n z e r o n u m b e r c , a n e w ( s y m m e t r i c ) c o n n e c t i o n
i s d e f i n e d
by
v = v + c D 2)
w h i c h m e a n s t h a t g i v e n v e c t o r f i e l d s X a n d Y,
v
χ
Y = v
χ
Y + c D ( X , Y )
w h e r e
9
( D ( X , Y ) , Z ) Ξ D ( X , Y , Z )
f o r al l v e c t o r f i e l d s Z. N o w , w h e n M is a f a m i l y of d e n s i t i e s a n d g a n d D a r e
d e f i n e d ,
in
t e r m s
of an
a r b i t r a r y p a r a m e t e r i z a t i o n ,
as
8/18/2019 Differential Geometry in Statistical Inference
16/246
10 Robert E. Kass
w h e r e
is t h e l o g l i k e l i h o o d f u n c t i o n , a n d if c = α / 2 , t h e n ( 2 ) d e f i n e s t h e
α c o n n e c t i o n .
In t h i s s t a t i s t i c a l c a s e ,
D is
n o t o n l y s y m m e t r i c
in
i t s f i r s t t w o
a r g u m e n t s ,
as it
m u s t
be in
( 2 ) ,
it is
s y m m e t r i c
in
a l l t h r e e . L a u r i t z e n
t h e r e f o r e d e f i n e s an a b s t r a c t s t a t i s t i c a l m a n i f o l d to be a t r i p l e ( M , g , D ) in
w h i c h M is a s m o o t h m d i m e n s i o n a l m a n i f o l d , g is a R i e m a n n i a n m e t r i c , a n d D is
a c o m p l e t e l y s y m m e t r i c c o v a r i a n t 3 t e n s o r . W i t h t h i s a d d i t i o n a l s y m m e t r y
c o n s t r a i n t a l o n e , he t h e n p r o c e e d s to e s t a b l i s h a l a r g e n u m b e r of b a s i c p r o p e r
t i e s ,
e s p e c i a l l y t h o s e r e l a t i n g
to
t h e d u a l i t y s t r u c t u r e A m a r i d e s c r i b e d .
The
t r e a t m e n t
is
f u l l y g e o m e t r i c a l
1 1
or
c o o r d i n a t e f r e e . T h i s
is
a e s t h e t i c a l l y
a p p e a l i n g , e s p e c i a l l y to t h o s e w h o l e a r n e d l i n e a r m o d e l s in t h e c o o r d i n a t e f r e e
s e t t i n g . L a u r i t z e n ' s p r i m a r y p u r p o s e is to s h o w t h a t t h e a p p r o p r i a t e m a t h e m a t
i c a l o b j e c t of s t u d y is o n e t h a t is n o t p a r t of t h e s t a n d a r d d i f f e r e n t i a l
g e o m e t r y , b u t d o e s h a v e m a n y s p e c i al f e a t u r e s a r i s i n g f r o m
an
a p p a r e n t l y s i m p l e
s t r u c t u r e .
He
n o t o n l y p r e s e n t s t h e a b s t r a c t g e n e r a l i t i e s a b o u t α c o n n e c t i o n s
o n s t a t i s t i c a l m a n i f o l d s , he a l s o e x a m i n e s f i v e e x a m p l e s in f u ll d e t a i l . The
f i r s t is t h e u n i v a r i a t e G a u s s i a n m o d e l , t h e s e c o n d is t h e i n v e r s e G a u s s i a n
m o d e l , t h e t h i r d is t h e t w o p a r a m e t e r g a m ma m o d e l , a n d t h e l a s t t w o a r e
s p e c i a l l y c o n s t r u c t e d m o d e l s t h a t d i s p l a y i n t e r e s t i n g p o s s i b i l i t i e s
of
t h e n o n
s t a n d a r d g e o m e t r i e s
of
α c o n n e c t i o n s .
In
p a r t i c u l a r , t h e l a t t e r t w o s t a t i s t i c a l
m a n i f o l d s a r e n o t w h a t L a u r i t z e n c a l l s c o n j u g a t e s y m m e t r i c a n d so t h e
s e c t i o n a l c u r v a t u r e s do n o t d e t e r m i n e t h e R i e m a n n t e n s o r ( as t h e y do in
R i e m a n n i a n
g e o m e t r y ) .
He
a l s o d i s c u s s e s t h e c o n s t r u c t i o n
of
g e o d e s i c f o l i a
t i o n s ,
w h i c h a r e d e c o m p o s i t i o n s
of
t h e m a n i f o l d a n d a r e i m p o r t a n t b e c a u s e t h e y
g e n e r a t e p o t e n t i a l l y i n t e r e s t i n g d e c o m p o s i t i o n s
of
t h e s a m p l e s p a c e .
At
t h e
e n d of h i s p a p e r , L a u r i t z e n c a l l s a t t e n t i o n to s e v er a l o u t s t a n d i n g p r o b l e m s .
A m a r i ' s α c o n n e c t i o n s , b a s e d on t h e f i r s t t h r e e m o m e n t s of t h e
s c o r e f u n c t i o n , do n o t f u r n i s h t h e o n l y e x a m p l e s of s t a t i s t i c a l m a n i f o l d s . In
h i s c o n t r i b u t i o n to t h i s v o l u m e , B a r n d o r f f N i e l s e n p r e s e n t s a n o t h e r c l a s s of
e x a m p l e s b a s e d i n s t e a d
on
c e r t a i n o b s e r v e d r a t h e r t h a n e x p e c t e d d e r i v a t i v e s
8/18/2019 Differential Geometry in Statistical Inference
17/246
Introduction 11
o f t h e l o g l i k e l i h o o d .
A l t h o u g h t h e i d e a of u s i n g o b s e r v e d d e r i v a t i v e s m i g h t o c c u r to
a n y c a s u a l l i s t e n e r
on
b e i n g t o l d
of
A m a r i ' s u s e
of
e x p e c t a t i o n s ,
it is not
o b v i o u s h o w to i m p l e m e n t i t . F i r s t of a l l , in o r d e r to d e f i n e an o b s e r v e d
i n f o r m a t i o n R i e m a n n i a n m e t r i c , o n e n e e d s a d e f i n i t i o n of o b s e r v e d i n f o r m a t i o n
a t e a c h p o i n t of t h e p a r a m e t e r s p a c e . A p p a r e n t l y o n e w o u l d w a n t to t r e a t e a c h
θ
as if it w e r e an M L E a n d t h e n u s e I ( θ ) . H o w e v e r , I ( θ ) d e p e n d s on t h e w h o l e
s a m p l e y r a t h e r t h a n on θ a l o n e , so t h i s s c h e m e d o e s n o t y e t p r o v i d e an e x p l i c i t
d e f i n i t i o n . B a r n d o r f f N i e l s e n ' s s o l u t i o n is n a t u r a l in t h e c o n t e x t of h i s
r e s e a r c h
on
c o n d i t i o n a l i t y :
he
r e p l a c e s t h e s a m p l e
y
w i t h
a
s u f f i c i e n t p a i r
( θ , a )
w h e r e a is t h e o b s e r v e d v a l u e of an a s y m p t o t i c a l l y a n c i l l a r y s t a t i s t i c A.
T h i s is a l w a y s p o s s i b l e f o r c u r v e d e x p o n e n t i a l f a m i l i e s , and in m o r e g e n e ra l
m o d e l s A c o u l d at
l e a s t
be t a k e n so t h a t ( θ , A ) is a s y m p t o t i c a l l y s u f f i c i e n t .
W i t h t h i s r e p l a c e m e n t , t h e s e c o n d c o m p o n e n t m a y be h e l d f i x e d at A = a w h i l e θ
v a r i e s .
W r i t i n g I ( θ ) = I,g χ ( θ ) t h u s a l l o w s t h e d e f i n i t i o n I ( θ ) Ξ I, JQ
t o be m a d e at e a c h p o i n t θ in t h e p a r a m e t e r s p a c e . U s i n g t h i s d e f i n i t i o n of
t h e R i e m a n n i a n m e t r i c , B a r n d o r f f N i e l s e n d e r i v e s t h e c o e f f i c i e n t s t h a t d e t e r
m i n e t h e R i e m a n n i a n c o n n e c t i o n . F r o m t h e t r a n s f o r m a t i o n p r o p e r t i e s of t e n s o r s ,
h e t h e n s u c c e e d s in f i n d i n g an a n a l o g u e of t h e e x p o n e n t i a l c o n n e c t i o n b a s e d on
a c e r t a i n m i x e d t h i r d d e r i v a t i v e of t h e l o g l i k e l i h o o d f u n c t i o n ( t w o d e r i v a t i v e s
b e i n g t a k e n w i t h r e s p e c t to θ as p a r a m e t e r , o n e w i t h r e s p e c t to θ as M L E ) . In
s o d o i n g , he d e f i n e s t h e t e n s o r D in t h e s t a t i s t i c a l m a n i f o l d a n d t h u s a r r i v e s
a t h i s o b s e r v e d c o n d i t i o n a l g e o m e t r y .
B a r n d o r f f N i e l s e n ' s i n t e r e s t in t h i s g e o m e t r y l i e s n o t w i t h
a n a l o g u e s of s t a t i s t i c a l c u r v a t u r e a n d o t h e r e x p e c t e d g e o m e t r y c o n s t r u c t s , but
r a t h e r w i t h an a l t e r n a t i v e d e r i v a t i o n , i n t e r p r e t a t i o n , a n d e x t e n s i o n of an
a p p r o x i m a t i o n to t h e c o n d i t i o n a l d e n s i t y of t h e M L E , w h i c h h a d b e e n o b t a i n e d
e a r l i e r ( in B a r n d o r f f N i e l s e n a n d C o x , 1 9 7 9 ) . In s e v e r a l p a p e r s , B a r n d o r f f
N i e l s e n ( 1 9 8 0 , 1 9 8 3 ) h a s d i s c u s s e d g e n e r a l i z a t i o n s a n d a p p r o x i m a t e v e r s i o n s of
F i s h e r ' s f u n d a m e n t a l d e n s i t y l i k e l i h o o d f o r m u l a f o r l o c a t i o n m o d e l s
8/18/2019 Differential Geometry in Statistical Inference
18/246
Robert E. Kass
p θ |a,θ) = c L θ)/L θ) 3)
w h e r e θ is t h e N I LE , a is an a n c i l l a r y s t a t i s t i c , p is t h e c o n d i t i o n a l d e n s i t y
o f t h e N IL E, a n d
L is
t h e l i k e l i h o o d f u n c t i o n . ( T h i s
is
d i s c u s s e d
in
E f r o n a n d
H i n k l e y , 1 9 7 8 ; F i s h e r a c t u a l l y t r e a t e d t h e l o c a t i o n s c a l e c a s e . ) T h e f o r m u l a
i s of g r e a t i m p o r t a n c e b o t h p r a c t i c a l l y , s i n c e it p r o v i d e s a w a y of c o m p u t i n g
t h e c o n d i t i o n a l d e n s i t y , a n d p h i l o s o p h i c a l l y , s i n c e it e n t a i l s t h e f o r m a l
a g r e e m e n t of c o n d i t i o n a l i n f e r e n c e a n d B a y e s i a n i n f e r e n c e u s i n g an i n v a r i a n t
p r i o r . I n s p e c t i o n of t h e d e r i v a t i o n i n d i c a t e s
t h a t
t h e f o r m u l a r e s u l t s f r o m
t h e t r a n s f o r m a t i o n a l n a t u r e of t h e l o c a t i o n p r o b l e m , a n d B a r n d o r f f N i e l s e n h a s
s h o w n t h a t
a
v e r s i o n
of it
( w i t h
an
a d d i t i o n a l f a c t o r f o r t h e v o l u m e e l e m e n t )
h o l d s f o r y e r y g e n e r a l t r a n s f o r m a t i o n m o d e l s . He h a s a l s o s h o w n
t h a t
f o r n o n
t r a n s f o r m a t i o n m o d e l s , a v e r s i o n of t h e r i g h t h a n d s i d e of ( 3 ) w h i l e n o t
e x a c t l y e q u a l to t h e l e f t h a n d s i d e , r e m a i n s a g o o d a s y m p t o t i c a p p r o x i m a t i o n f o r
i t . ( S e e a l s o H i n k l e y , 1 9 8 0 , a n d N l c Cu l l a g h , 1 9 8 4 a . ) In h i s p a p e r in t h i s
v o l u m e , B a r n d o r f f N i e l s e n r e v i e w s t h e s e r e s u l t s , s h o w s h o w t h e v a r i o u s o b s e r v e d
c o n d i t i o n a l g e o m e t r i c a l q u a n t i t i e s a r e c a l c u l a t e d , an d t h e n d e r i v e s h i s d e s i r e d
e x p a n s i o n ( o f a v e r s i o n of t h e r i g h t h a n d s i d e of ( 3 ) ) in t e r m s of t h e g e o
m e t r i c a l q u a n t i t i e s
t h a t
c o r r e s p o n d to t h o s e u s e d by A m a r i in h i s e x p e c t e d
g e o m e t r y e x p a n s i o n s . B a r n d o r f f N i e l s e n d e v o t e s s u b s t a n t i a l a t t e n t i o n to t r a n s
f o r m a t i o n m o d e l s , w h i c h m a y be t r e a t e d w i t h i n h i s f r a m e w o r k of o b s e r v e d
c o n d i t i o n a l g e o m e t r y . In t h i s c o n t e x t , t h e m o d e l s b e c o m e L i e G r o u p s , f o r w h i c h
t h e r e is a r i c h m a t h e m a t i c a l t h e o r y .
I n t h e f o u r t h p a p e r in t h i s v o l u m e , P r o f e s s o r R a o r e t u r n s to t h e
c h a r a c t e r i z a t i o n of t h e i n f o r m a t i o n m e t r i c
t h a t
o r i g i n a l l y l e d h i m ( an d a l s o
J e f f r e y s ) to i n t r o d u c e i t : it is an i n f i n i t e s i m a l m e a s u r e of d i v e r g e n c e b a s e d
o n
w h a t
is n o w c a l l e d S h a n n o n e n t r o p y . R a o c o n s i d e r s h e r e a m o r e g e n e r a l c l a s s
o f d i v e r g e n c e m e a s u r e s , w h i c h he h a s f o u n d u s e f u l in t h e s t u d y of g e n e t i c
d i v e r s i t y , l e a d i n g to a w i d e v a r i e t y of m e t r i c s . He d e r i v e s t h e q u a d r a t i c a n d
c u b i c t e r m s
in
T a y l o r s e r i e s e x p a n s i o n s
of
t h e s e m e a s u r e s a n d s h o w s h o w ,
in
t h e
c as e o f S h a n n o n e n t r o p y , t h e c u b i c t e r m is r e l a t e d to t h e α c o n n e c t i o n s .
8/18/2019 Differential Geometry in Statistical Inference
19/246
Introduction
13
T h e p a p e r s h e r e c o l l e c t i v e l y s h o w t h a t g e o m e t r i c a l s t r u c t u r e s of
s t a t i s t i c a l m o d e l s c a n p r o v i d e b o t h c o n c e p t u a l s i m p l i f i c a t i o n s a n d n e w m e t h o d s
o f a n a l y s i s f o r p r o b l e m s
of
s t a t i s t i c a l i n f e r e n c e . T h e r e
is
i n t e r e s t i n g
m a t h e m a t i c s i n v o l v e d , b u t d o e s t h e i n t e r e s t i n g m a t h e m a t i c s l e ad
to
i n t e r e s t i n g
s t a t i s t i c s ?
T h e q u e s t i o n a r i s e s b e c a u s e g e o m e t r y h a s p r o v i d e d n e w t e c h n i q u e s ,
a n d i t s f o r m a l i s m p r o d u c e s c o n v e n i e n t s u m m a r i e s f o r c o m p l i c a t e d m u l t i v a r i a t e
e x p r e s s i o n s in a s y m p t o t i c e x p a n s i o n s ( as in A m a r i and K u m o n , 1 9 8 3 , and
M c C u l l a g h ,
1 9 8 4 b ) ,
b u t it h a s n o t y e t c r e a t e d n e w m e t h o d o l o g y w i t h c l e a r l y
i m p o r t a n t p r a c t i c a l a p p l i c a t i o n s . T h u s ,
it is
a l r e a d y a p p a r e n t f r o m ( 1 )
t h a t
t h e r e e x i s t s
a
w i d e c l a s s
of
e s t i m a t o r s t h a t m i n i m i z e i n f o r m a t i o n l o s s ( a nd a r e
s e c o n d o r d e r e f f i c i e n t ) : it c o n s i s t s of t h o s e h a v i n g z e r o m i x t u r e c u r v a t u r e
f o r t h e i r a s s o c i a t e d a n c i l l a r y f a m i l i e s . It is i n t e r e s t i n g t h a t t h e M L E is o n l y
o n e m e m b e r of t h i s c l a s s , a n d it is n i c e to h a v e E g u c h i ' s ( 1 9 8 3 ) d e r i v a t i o n t h a t
c e r t a i n m i n i m u m c o n t r a s t e s t i m a t o r s a r e o t h e r m e m b e r s , b u t
it
s e e m s u n l i k e l y
t h o u g h a d m i t t e d l y p o s s i b l e
t h a t a n y c o m p e t i t o r w i l l r e p l a c e m a x i m u m l i k e l i h o o d
e s t i m a t i o n as t h e p r i m a r y m e t h o d of c h o i c e in p r a c t i c e . S i m i l a r l y , t h e r e is
n o t y e t a n y r e a s o n to t h i n k t h a t a l t e r n a t i v e m i n i m u m α d i v e r g e n c e e s t i m a t o r s or
t h e i r o b s e r v e d c o n d i t i o n a l g e o m e t r y c o u n t e r p a r t s w i l l be c o n s i d e r e d s u p e r i o r to
t h e M L E .
O n t h e o t h e r h a n d , as I i n d i c a t e d at t h e o u t s e t , g e o m e t r y d o e s
g i v e a d e f i n i t i v e d e s c r i p t i o n of i n f o r m a t i o n l o s s a n d r e c o v e r y . S i n c e F i s h e r
r e m a i n s o u r w i s e s t y e t
m o s t
e n i g m a t i c s a g e ,
it is
w o r t h o u r w h i l e
to
t r y
to
u n d e r s t a n d h i s p r o n o u n c e m e n t s . T o g e t h e r w i t h t h e t r i u m v i r a t e
of
c o n s i s t e n c y ,
**
S i n c e R a o ' s w o r k
on
s e c o n d o r d e r e f f i c i e n c y a r o s e
in an
a t t e m p t
to
u n d e r s t a n d
F i s h e r ' s c o m p u t a t i o n
of
i n f o r m a t i o n l o s s
in
e s t i m a t i o n ,
it
m i g h t a p p e a r t h a t
E f r o n ' s i n v e s t i g a t i o n a l s o b e g a n
as an
a t t e m p t
to
u n d e r s t a n d F i s h e r .
He
h a s
i n f o r m e d m e , h o w e v e r , t h a t he s e t o u t to d e f i n e t h e c u r v a t u r e of a s t a t i s t i c a l
m o d e l a n d c a m e l a t e r
to
i t s u s e
in
i n f o r m a t i o n l o s s a n d s e c o n d o r d e r e f f i c i e n c y .
8/18/2019 Differential Geometry in Statistical Inference
20/246
1 4 R o b e r t E . K a s s
s u f f i c i e n c y , a n d e f f i c i e n c y , i n f o r m a t i o n l o s s a n d r e c o v e r y f o r m t h e c o r e o f
F i s h e r ' s t h e o r y o f e s t i m a t i o n . O n t h e b a s i s o f t h e g e o m e t r i c a l r e s u l t s , i t i s
f a i r t o s a y t h a t w e n o w k n o w w h a t F i s h e r w a s t a l k i n g a b o u t , a n d t h a t w h a t h e
s a i d w a s t r u e . H e r e , a s i n o t h e r p r o b l e m s ( s u c h a s i n f e r e n c e w i t h n u i s a n c e
p a r a m e t e r s , d i s c u s s e d i n A m a r i ' s s e c t i o n 5 , o r i n n o n l i n e a r r e g r e s s i o n , e . g . ,
B a t e s a n d W a t t s , 1 9 8 0 , C o o k a n d T s a i , 1 9 8 5 , K a s s , 1 9 8 4 , M c C u l l a g h a n d C o x , 1 9 3 6 ) ,
t h e g e o m e t r i c a l f o r m u l a t i o n t e n d s t o s h i f t t h e b u r d e n o f d e r i v a t i o n o f r e s u l t s
a w a y f r o m p r o o f s , t o w a r d d e f i n i t i o n s . T h u s , o n c e t h e s t a t e m e n t o f a p r o p o s i t i o n
i s u n d e r s t o o d , i t s t r u t h i s e a s i e r t o s e e a n d i n t h i s t h e r e i s g r e a t s i m p l i f i c a -
t i o n . O n e c o u l d m a k e t h i s a r g u m e n t a b o u t m u c h a b s t r a c t m a t h e m a t i c a l d e v e l o p -
m e n t , b u t i t i s p a r t i c u l a r l y a p p r o p r i a t e h e r e .
F u r t h e r m o r e , t h e r e a r e r e a s o n s t o t h i n k t h a t f u t u r e w o r k i n t h i s
a r e a c o u l d l e a d t o u s e f u l r e s u l t s t h a t w o u l d o t h e r w i s e b e d i f f i c u l t t o o b t a i n .
O n e i m p o r t a n t p r o b l e m t h a t s t r u c t u r a l r e s e a r c h m i g h t s o l v e i s t h a t o f d e t e r m i n -
i n g u s e f u l c o n d i t i o n s u n d e r w h i c h a p a r t i c u l a r r o o t o f t h e l i k e l i h o o d e q u a t i o n
w i l l a c t u a l l y m a x i m i z e t h e l i k e l i h o o d . G l o b a l r e s u l t s o n f o l i a t i o n s m i g h t b e
v e r y h e l p f u l , a s m i g h t b e f o r m u l a s r e l a t i n g c o m p u t a b l e c h a r a c t e r i s t i c s o f
s t a t i s t i c a l m a n i f o l d s t o t h e b e h a v i o r o f g e o d e s i e s . T h e r e s u l t s i n t h e s e p a p e r s
c o u l d t u r n o u t t o p l a y a c e n t r a l r o l e i n t h e s o l u t i o n o f t h i s o r s o m e o t h e r
p r a c t i c a l p r o b l e m o f s t a t i s t i c a l t h e o r y . W e w i l l h a v e t o w a i t a n d s e e . U n t i l
t h e n , r e a d e r s m a y e n j o y t h e p a p e r s a s i n f o r m a t i v e e x c u r s i o n s i n t o a n i n t r i g u i n g
r e a l m o f m a t h e m a t i c a l s t a t i s t i c s .
A c k n o w l e d g e m e n t s
I t h a n k 0 . E . B a r n d o r f f - N i e l s e n , D . R . C o x , a n d C . R . R a o f o r t h e i r
c o m m e n t s o n a n e a r l i e r d r a f t . T h i s p a p e r w a s p r e p a r e d w i t h s u p p o r t f r o m t h e
N a t i o n a l S c i e n c e F o u n d a t i o n u n d e r G r a n t N o . N S F / D M S - 8 5 0 3 0 1 9 .
8/18/2019 Differential Geometry in Statistical Inference
21/246
R E F E R E N C E S
A m a r i , S. ( 1 9 8 2 a ) . D i f f e r e n t i a l g e o m e t r y of c u r v e d e x p o n e n t i a l f a m i l i e s -
c u r v a t u r e s
and
i n f o r m a t i o n l o s s . A n n . S t a t i s t . 1 0 , 3 5 7 - 3 8 7 .
A m a r i ,
S.
( 1 9 8 2 b ) . G e o m e t r i c a l t h e o r y
of
a s y m p t o t i c a n c i l l a r i t y
and
c o n d i t i o n a l
i n f e r e n c e . B i o m e t r i k a 6 9 , 1 - 1 7 .
A m a r i ,
S. and
K u m o n ,
M.
( 1 9 8 3 ) .
D i f f e r e n t i a l g e o m e t r y
of
E d g e w o r t h e x p a n s i o n s
i n c u r v e d e x p o n e n t i a l f a m i l y . A n n . I n s t . S t a t i s t . M a t h . 3 5 A , 1 - 2 4 .
B a r n d o r f f - N i e l s e n ,
0. E. ( 1 9 7 8 ) .
I n f o r m a t i o n
and
E x p o n e n t i a l F a m i l i e s ,
N e w Y o r k : W i l e y .
B a r n d o r f f - N i e l s e n ,
0. E. ( 1 9 8 0 ) .
C o n d i t i o n a l i t y r e s o l u t i o n s . B i o m e t r i k a
67,
2 9 3 - 3 1 0 .
B a r n d o r f f - N i e l s e n , 0. E. ( 1 9 8 3 ) . On a f o r m u l a for he d i s t r i b u t i o n of the
m a x i m u m l i k e l i h o o d e s t i m a t o r . B i o m e t r i k a 7 0 , 3 4 3 - 3 0 5 .
B a r n d o r f f - N i e l s e n , 0. E. and C o x , D. R. ( 1 9 7 9 ) . E d g e w o r t h and S a d d l e p o i n t
a p p r o x i m a t i o n s w i t h s t a t i s t i c a l a p p l i c a t i o n s , ( w i t h D i s c u s s i o n ) .
J .
R.
S t a t i s t . S o c . B 4 1 , 2 7 9 - 3 1 2 .
B a t e s ,
D. M. and a t t s , D. G. ( 1 9 8 0 ) . R e l a t i v e c u r v a t u r e m e a s u r e s of n o n -
l i n e a r i t y . J. R. S t a t i s t . S o c . B 4 2 , 1 - 2 5 .
B e r a n ,
R.
( 1 9 7 7 ) .
M i n i m u m H e l l i n g e r d i s t a n c e e s t i m a t e s f o r p a r a m e t r i c m o d e l s .
A n n . S t a t i s t . 5, 4 4 5 - 4 6 3 .
C e n t s o v ,
N. N. ( 1 9 7 1 ) .
S t a t i s t i c a l D e c i s i o n R u l e s
and
O p t i m a l I n f e r e n c e
in
R u s s i a n ) . T r a n s l a t e d i n t o E n g l i s h
( 1 9 8 2 ) ,
A M S , R h o d e I s l a n d .
C o o k , R. D. and s a i , C . - L . ( 1 9 8 5 ) . R e s i d u a l s in n o n l i n e a r r e g r e s s i o n .
iometrik 72 23-29.
8/18/2019 Differential Geometry in Statistical Inference
22/246
16
Robert E Kass
C o x ,
D. R. ( 1 9 3 0 ) .
L o c a l a n c i l l a r i t y . B i o m e t r i k a 6 2 , 2 6 9 - 2 7 6 .
D a w i d , A. P. ( 1 9 7 5 ) . D i s c u s s i o n to E f r o n ' s p a p e r . A n n . S t a t i s t . 3, 1 2 3 1 - 1 2 3 4 .
E f r o n , B. ( 1 9 7 5 ) . D e f i n i n g t h e c u r v a t u r e of a s t a t i s t i c a l p r o b l e m ( w i t h
a p p l i c a t i o n s to s e c o n d - o r d e r e f f i c i e n c y ) , ( w i t h D i s c u s s i o n ) .
A n n . S t a t i s t .
3,
1 1 8 9 - 1 2 4 2 .
E f r o n ,
B. ( 1 9 7 8 ) .
T h e g e o m e t r y
of
e x p o n e n t i a l f a m i l i e s . A n n . S t a t i s t .
6,
3 6 2 - 3 7 6 .
E f r o n , B. and i n k l e y , D. V. ( 1 9 7 8 ) . A s s e s s i n g t h e a c c u r a c y of t h e m a x i m u m
l i k e l i h o o d e s t i m a t o r : O b s e r v e d v e r s u s e x p e c t e d F i s h e r i n f o r m a t i o n ,
( w i t h
d i s c u s s i o n ) .
B i o m e t r i k a 6 5 , 4 5 7 - 4 8 7 .
E g u c h i ,
S. ( 1 9 8 3 ) .
S e c o n d o r d e r e f f i c i e n c y
of
m i n i m u m c o n t r a s t e s t i m a t o r s
in
a c u r v e d e x p o n e n t i a l f a m i l y . A n n . S t a t i s t . 1 1 , 7 9 3 - 8 0 3 .
F i s h e r , R. A. ( 1 9 2 5 ) . T h e o r y of s t a t i s t i c a l e s t i m a t i o n . P r o c . C a m b . P h i l . Soc.
22^, 7 0 0 - 7 2 5 .
F i s h e r , R. A.
( 1 9 3 4 ) .
T w o n e w p r o p e r t i e s of m a t h e m a t i c a l l i k e l i h o o d . P r o c .
R . S o c . A 1 4 4 , 2 8 5 - 3 0 7 .
G h o s h ,
J. K. and
u b r a m a n i a m ,
K. ( 1 9 7 4 ) .
S e c o n d o r d e r e f f i c i e n c y
of
m a x i m u m
l i k e l i h o o d e s t i m a t o r s . S a n k y a 3 6 A , 3 2 5 - 3 5 8 .
H i n k l e y ,
D. V. ( 1 9 8 0 ) .
L i k e l i h o o d
as
a p p r o x i m a t e p i v o t a l d i s t r i b u t i o n .
B i o m e t r i k a 6 7 , 2 8 7 - 2 9 2 .
J e f f r e y s , H. ( 1 9 4 6 ) . An i n v a r i a n t f o r m f o r t h e p r i o r p r o b a b i l i t y in e s t i m a t i o n
p r o b l e m s . P r o c . R o y . S o c . A 8 6 , 4 5 3 - 4 6 1 .
K a s s ,
R. E.
( 1 9 8 4 ) .
C a n o n i c a l p a r a m e t e r z a t i o n s
and
e r o p a r a m e t e r - e f f e c t s
c u r v a t u r e . J. R o y . S t a t i s t . S o c . B 4 6 , 1,
8 6 - 9 2 .
M a d s e n ,
L. T. ( 1 9 7 9 ) .
T h e g e o m e t r y
of
s t a t i s t i c a l m o d e l
- a
g e n e r a l i z a t i o n
of
c u r v a t u r e . R e s . R e p o r t 7 9 - 1 . S t a t i s t . R e s . U n i t , D a n i s h M e d i c a l
R e s .
C o u n c i l .
M c C u l i a g h , P. ( 1 9 8 4 a ) . On l o c a l s u f f i c i e n c y . B i o m e t r i k a 7 1 , 2 3 3 - 2 4 4 .
M c C u l l a g h , P. ( 1 9 8 4 b ) . T e n s o r n o t a t i o n a n d c u m u l a n t s of p o l y n o m i a l s .
B i o m e t r i k a 7 1 , 4 6 1 - 4 7 6 .
8/18/2019 Differential Geometry in Statistical Inference
23/246
ntrodu tion l
McCullagh,
P
and Cox,
D R
1986 ) . Invar iants and l ike l ihood ra t io s ta t is t ic s .
n n
S ta t i s t . 14 , 1419-1430.
P i e r c e ,
D. A. ( 1 9 7 5 ) .
D i s c u s s i o n
to
E f r o n ' s p a p e r . A n n . S t a t i s t .
3,
1 2 1 9 - 1 2 2 1 .
R a o ,
C. R. ( 1 9 4 5 ) . I n f o r m a t i o n a n d a c c u r a c y a t t a i n a b l e in t h e e s t i m a t i o n of
s t a t i s t i c a l p a r a m e t e r s . B u l l . C a l c u t t a M a t h . S o c . 3 7 , 8 1 - 8 9 .
R a o , C. R. ( 1 9 6 1 ) . A s y m p t o t i c e f f i c i e n c y a n d l i m i t i n g i n f o r m a t i o n . P r o c .
F o u r t h B e r k e l e y S y m p . M a t h . S t a t i s t . P r o b . , E d i t e d by J. N e y m a n ,
1 , 5 3 1 - 5 4 5 .
R a o ,
C. R. ( 1 9 6 2 ) . E f f i c i e n t e s t i m a t e s and o p t i m u m i n f e r e n c e p r o c e d u r e s in
l a r g e s a m p l e s ( w i t h d i s c u s s i o n ) .
J.
R o y . S t a t i s t . S o c . B 2 4 , 4 6 - 7 2 .
R a o , C. R. ( 1 9 6 3 ) . C r i t e r i a of e s t i m a t i o n in l a r g e s a m p l e s . S a n k y a 2 5 , 1 8 9 -
2 0 6 .
R e e d s , J. ( 1 9 7 5 ) . D i s c u s s i o n to E f r o n ' s p a p e r . A n n . S t a t i s t . 3, 1 2 3 4 - 1 2 3 8 .
S k o v g a a r d , I.
( 1 9 8 5 ) .
s e c o n d - o r d e r i n v e s t i g a t i o n of a s y m p t o t i c a n c i i l a r i t y .
A n n . S t a t i s t . 1 3 , 5 3 4 - 5 5 1 .
8/18/2019 Differential Geometry in Statistical Inference
24/246
8/18/2019 Differential Geometry in Statistical Inference
25/246
D I F F E R E N T I A L G E O M E T R I C A L T H E O R Y O F S T A T I S T I C S
S h u n - i c h i A m a r i *
1 . I n t r o d u c t i o n 2 1
2 .
G e o m e t r i c a l S t r u c t u r e o f S t a t i s t i c a l M o d e l s 2 5
3 . H i g h e r - O r d e r A s y m p t o t i c T h e o r y o f S t a t i s t i c a l I n f e r e n c e i n
C u r v e d E x p o n e n t i a l F a m i l y 3 8
4 . I n f o r m a t i o n , S u f f i c i e n c y a n d A n c i l l a r i t y H i g h e r O r d e r T h e o r y 5 2
5 . F i b r e - B u n d l e T h e o r y o f S t a t i s t i c a l M o d e l s 5 9
6 . E s t i m a t i o n o f S t r u c t u r a l P a r a m e t e r i n t h e P r e s e n c e o f I n f i n i t e l y
M a n y N u i s a n c e P a r a m e t e r s 7 3
7 .
P a r a m e t r i c M o d e l s o f S t a t i o n a r y G a u s s i a n T i m e S e r i e s 8 3
8 . R e f e r e n c e s 9 1
D e p a r t m e n t o f M a t h e m a t i c a l E n g i n e e r i n g a n d I n s t r u m e n t a t i o n P h y s i c s , U n i v e r s i t y
o f T o k y o , T o k y o , J A P AN
19
8/18/2019 Differential Geometry in Statistical Inference
26/246
8/18/2019 Differential Geometry in Statistical Inference
27/246
1 . I N T R O D U C T I O N
S t a t i s t i c s i s a s c i e n c e w h i c h s t u d i e s m e t h o d s o f i n f e r e n c e , f r o m
o b s e r v e d d a t a , c o n c e r n i n g t h e p r o b a b i l i s t i c s t r u c t u r e u n d e r l y i n g s u c h d a t a .
T h e c l a s s o f a ll t h e p o s s i b l e p r o b a b i l i t y d i s t r i b u t i o n s i s u s u a l l y t o o w i d e t o
c o n s i d e r a l l i t s e l e m e n t s a s c a n d i d a t e s f o r t h e t r u e p r o b a b i l i t y d i s t r i b u t i o n
f r o m w h i c h t h e d a t a w e r e d e r i v e d . S t a t i s t i c i a n s o f t e n a s s u m e a s t a t i s t i c a l
m o d e l w h i c h i s a s u b s e t o f t h e s e t o f al l t h e p o s s i b l e p r o b a b i l i t y d i s t r i b u -
t i o n s ,
a n d e v a l u a t e p r o c e d u r e s o f s t a t i s t i c a l i n f e r e n c e a s s u m i n g t h a t t h e m o d e l
i s f a i t h f u l , i . e . , it i n c l u d e s t h e t r u e d i s t r i b u t i o n . I t s h o u l d , h o w e v e r , b e
r e m a r k e d t h a t a m o d e l i s n o t n e c e s s a r i l y f a i t h f u l b u t is a p p r o x i m a t e l y s o . I n
e i t h e r c a s e , i t s h o u l d b e v e r y i m p o r t a n t t o k n o w t h e s h a p e o f a s t a t i s t i c a l
m o d e l i n t h e w h o l e s e t o f p r o b a b i l i t y d i s t r i b u t i o n s . T h i s i s t h e g e o m e t r y o f a
s t a t i s t i c a l m o d e l . A s t a t i s t i c a l m o d e l o f t e n f o r m s a g e o m e t r i c a l m a n i f o l d , s o
t h a t t h e g e o m e t r y o f m a n i f o l d s s h o u l d p l a y a n i m p o r t a n t r o l e . C o n s i d e r i n g t h a t
p r o p e r t i e s o f s p e c i f i c t y p e s o f p r o b a b i l i t y d i s t r i b u t i o n s , f o r e x a m p l e , o f
G a u s s i a n d i s t r i b u t i o n s , o f W i e n e r p r o c e s s e s , a nd s o o n , h a v e s o f a r b e e n s t u d i e d
i n d e t a i l , i t s e e m s r a t h e r s t r a n g e t h a t o n l y a f e w t h e o r i e s h a v e b e e n p r o p o s e d
c o n c e r n i n g p r o p e r t i e s o f a f a m i l y i t s e l f o f d i s t r i b u t i o n s . H e r e , by t h e p r o p e r -
t i e s o f a f a m i l y w e m e a n s u c h g e o m e t r i c r e l a t i o n s a s m u t u a l d i s t a n c e s , f l a t n e s s
o r c u r v a t u r e o f t h e f a m i l y , e t c . O b v i o u s l y i t i s n o t a t r i v i a l t a s k t o d e f i n e
s u c h g e o m e t r i c s t r u c t u r e s i n a n a t u r a l , u s e f u l a n d i n v a r i a n t m a n n e r .
O n l y l o ca l p r o p e r t i e s o f a s t a t i s t i c a l m o d e l a r e r e s p o n s i b l e f o r t h e
a s y m p t o t i c t h e o r y o f s t a t i s t i c a l i n f e r e n c e . L o c a l p r o p e r t i e s a r e r e p r e s e n t e d
b y t h e g e o m e t r y o f t h e t a n g e n t s p a c e s o f t h e m a n i f o l d . T h e t a n g e n t s p a c e h a s a
2 1
8/18/2019 Differential Geometry in Statistical Inference
28/246
2 2
Shun ichi Amari
n a t u r a l R i e m a n n i a n m e t r i c g i v e n by t h e F i s h e r i n f o r m a t i o n m a t r i x in the r e g u l a r
c a s e . It r e p r e s e n t s o n l y a l o c al p r o p e r t y of t h e m o d e l , b e c a u s e t h e t a n g e n t
s p a c e
is
n o t h i n g
but
lo c al l i n e a r i z a t i o n
of
t h e m o d e l m a n i f o l d .
In
o r d e r
to
o b t a i n l a r g e r - s c a l e p r o p e r t i e s ,
one
n e e d s
to
d e f i n e m u t u a l r e l a t i o n s
of the two
d i f f e r e n t t a n g e n t s p a c e s at t w o n e i g h b o r i n g p o i n t s in t h e m o d e l . T h i s can be
d o n e by d e f i n i n g a o n e - t o - o n e a f f i n e c o r r e s p o n d e n c e b e t w e e n two a n g e n t s p a c e s ,
w h i c h is c a l l e d an a f f i n e c o n n e c t i o n in d i f f e r e n t i a l g e o m e t r y . By an a f f i n e
c o n n e c t i o n , o n e c a n c o n s i d e r l o ca l p r o p e r t i e s a r o u n d e a c h p o i n t b e y o n d
the
l i n e a r a p p r o x i m a t i o n . T h e c u r v a t u r e
of a
m o d e l
can be
o b t a i n e d
by
t h e u s e
of
t h i s c o n n e c t i o n .
It is
c l e a r t h a t s u c h
a
d i f f e r e n t i a l - g e o m e t r i c a l c o n c e p t p r o -
v i d e s a t o ol c o n v e n i e n t f o r s t u d y i n g h i g h e r - o r d e r a s y m p t o t i c p r o p e r t i e s of
i n f e r e n c e . H o w e v e r , by c o n n e c t i n g l o ca l t a n g e n t s p a c e s f u r t h e r , o n e can o b t a i n
g l o b a l r e l a t i o n s . H e n c e , t h e v a l i d i t y of t h e d i f f e r e n t i a l - g e o m e t r i c a l m e t h o d is
n o t l i m i t e d w i t h i n t h e f r a m e w o r k
of
a s y m p t o t i c t h e o r y .
I t w a s
Rao
( 1 9 4 5 )
who
f i r s t p o i n t e d
out
h e i m p o r t a n c e
in the
d i f f e r e n t i a l - g e o m e t r i c a l a p p r o a c h .
He
i n t r o d u c e d
the
i e m a n n i a n m e t r i c
by
u s i n g
t h e F i s h e r i n f o r m a t i o n m a t r i x . A l t h o u g h a n u m b e r of r e s e a r c h e s h a v e b e e n
c a r r i e d o u t a l o n g t h i s R i e m a n n i a n l i n e ( s e e , e . g . , A m a r i ( 1 9 6 8 ) , A t k i n s o n and
M i t c h e l l ( 1 9 8 1 ) , D a w i d ( 1 9 7 7 ) , J a m e s ( 1 9 7 3 ) , K a s s ( 1 9 8 0 ) , S k o v g a a r d ( 1 9 8 4 ) ,
Y o s h i z a w a
( 1 9 7 1 ) , e t c . ) ,
t h e y
did not
a v e
a
l a r g e i m p a c t
on
s t a t i s t i c s . S o m e
a d d i t i o n a l c o n c e p t s
are
e c e s s a r y
to
i m p r o v e
its
u s e f u l n e s s .
A new
i d e a
was
d e v e l o p e d by C h e n t s o v ( 1 9 7 2 ) in h i s R u s s i a n b o o k ( a nd in s o m e p a p e r s p r i o r to
t h e
b o o k ) .
He
i n t r o d u c e d
a
f a m i l y
of
a f f i n e c o n n e c t i o n s
and
r o v e d t h e i r u n i q u e -
n e s s f r o m t h e p o i n t of v i e w of c a t e g o r i c a l i n v a r i a n c e . A l t h o u g h h i s t h e o r y w a s
d e e p and f u n d a m e n t a l , he did n o t d i s c u s s t h e c u r v a t u r e of a s t a t i s t i c a l m o d e l .
E f r o n ( 1 9 7 5 , 1 9 7 8 ) , i n d e p e n d e n t l y
of
C h e n t s o v
1
s w o r k , p r o v i d e d
a new
i d e a
by
p o i n t i n g out h a t t h e s t a t i s t i c a l c u r v a t u r e p l a y s an i m p o r t a n t r o l e in h i g h e r -
o r d e r p r o p e r t i e s of s t a t i s t i c a l i n f e r e n c e . D a w i d ( 1 9 7 5 ) p o i n t e d o u t f u r t h e r
p o s s i b i l i t i e s . E f r o n s i d ea w a s g e n e r a l i z e d by M a d s e n ( 1 9 7 9 ) ( s e e a l s o R e e d s
( 1 9 7 5 ) ) . A m a r i ( 1 9 8 0 , 1 9 8 2 a ) c o n s t r u c t e d a d i f f e r e n t i a l - g e o m e t r i c a l m e t h o d in
8/18/2019 Differential Geometry in Statistical Inference
29/246
Differential Geometrical Theory of Statistics 23
s t a t i s t i c s
by
i n t r o d u c i n g
a
f a m i l y
of
a f f i n e c o n n e c t i o n s , w h i c h h o w e v e r t u r n e d
o u t to be e q u i v a l e n t to C h e n t s o v ' s . He f u r t h e r d e f i n e d α c u r v a t u r e s , a n d p o i n t
e d o u t "he f u n d a m e n t a l r o l e s of t h e e x p o n e n t i a l a n d m i x t u r e c u r v a t u r e s p l a y e d in
s t a t i s t i c a l i n f e r e n c e . T h e t h e o r y h a s b e e n d e v e l o p e d f u r t h e r by a n u m b e r of
p a p e r s ( A m r n ( 1 9 8 2 b , 1 9 8 3 a , b ) , A m a r i
and
K u m o n
( 1 9 8 3 ) ,
K u m o n a nd A m a r i ( 1 9 8 3 ,
1 9 8 4 , 1 9 8 5 ) ,
N a g a o k a a n d A m a r i
( 1 9 8 2 ) ,
E g u c h i
( 1 9 8 3 ) ,
K a s s
( 1 9 8 4 ) ) .
T h e n e w
d e v e l o p m e n t s w e r e a l s o s h o w n
in
t h e N A T O R e s e a r c h W o r k s h o p
on
D i f f e r e n t i a l G e o
m e t r y in S t a t i s t i c a l I n f e r e n c e ( s e e B a r n d o r f f N i e l s e n ( 1 9 8 5 ) a n d L a u r i t z e n
( 1 9 8 5 ) ) . T h e y t o g e t h e r s e e m to p r o v e t h e u s e f u l n e s s of d i f f e r e n t i a l g e o m e t r y as
a f u n d a m e n t a l m e t h o d
in
s t a t i s t i c s . ( S ee a l s o C s i s z a r
( 1 9 7 5 ) ,
B u r b e a a n d R a o
( 1 9 8 2 ) ,
P f a n z a g l
( 1 9 8 2 ) ,
B e a l e
( 1 9 6 0 ) ,
B a t e s a n d W a t t s
( 1 9 8 0 ) ,
e t c . , f o r o t h e r
g e o m e t r i c a l w o r k . )
T h e p r e s e n t a r t i c l e g i v e s n o t o n l y
a
c o m p a c t r e v i e w
of
v a r i o u s
a c h i e v e m e n t s up to n o w by t h e d i f f e r e n t i a l g e o m e t r i c a l m e t h o d
m o s t
of w h i c h h a v e
a l r e a d y b e e n p u b l i s h e d in v a r i o u s j o u r n a l s and in A m a r i ( 1 9 8 5 ) b u t a l s o a p r e
v i e w of n e w r e s u l t s a n d h a l f b a k e d i d e a s in n e w d i r e c t i o n s ,
m o s t
of w h i c h h a v e
n o t y e t b e e n p u b l i s h e d . C h a p t e r
2
p r o v i d e s
an
i n t r o d u c t i o n
to
t h e g e o m e t r i c a l
m e t h o d , a n d e l u c i d a t e s f u n d a m e n t a l g e o m e t r i c a l p r o p e r t i e s
of
s t a t i s t i c a l m a n i
f o l d s . C h a p t e r
3 is
d e v o t e d
to
t h e h i g h e r o r d e r a s y m p t o t i c t h e o r y
of
s t a t i s t i
c al i n f e r e n c e , s u m m a r i z i n g h i g h e r o r d e r c h a r a c t e r i s t i c s of v a r i o u s e s t i m a t o r s
a n d t e s t s in g e o m e t r i c a l t e r m s . C h a p t e r 4 d i s c u s s e s a h i g h e r o r d e r t h e o r y of
a s y m p t o t i c s u f f i c i e n c y a n d a n c i 'l l ar i ty f r o m t h e F i s h e r i n f o r m a t i o n p o i n t
of
v i e w . R e f e r
to
A m a r i ( 1 9 8 5 ) f o r m o r e d e t a i l e d e x p l a n a t i o n s
in
t h e s e c h a p t e r s ;
L a u r i t z e n ( 1 9 8 5 ) g i v e s
a
g o o d i n t r o d u c t i o n
to
m o d e r n d i f f e r e n t i a l g e o m e t r y .
The
r e m a i n i n g C h a p t e r s 5, 6, a n d 7 t r e a t n e w i d e a s a n d d e v e l o p m e n t s w h i c h a r e
j u s t
u n d e r c o n s t r u c t i o n . In C h a p t e r 5 is i n t r o d u c e d a f i b r e b u n d l e a p p r o a c h , w h i c h
i s n e c e s s a r y
in
o r d e r
to
s t u d y p r o p e r t i e s
of
s t a t i s t i c a l i n f e r e n c e
in a
g e n e r a l
s t a t i s t i c a l m o d e l o t h e r t h a n
a
c u r v e d e x p o n e n t i a l f a m i l y .
A
H u b e r t b u n d l e a n d
a j e t b u n d l e a r e t r e a t e d
in a
g e o m e t r i c a l f r a m e w o r k
of
s t a t i s t i c a l i n f e r e n c e .
C h a p t e r 6 g i v e s a s u m m a r y of a t h e o r y of e s t i m a t i o n of a s t r u c t u r a l p a r a m e t e r
8/18/2019 Differential Geometry in Statistical Inference
30/246
24 Shun ichi Amari
i n t he p r e s e n c e of n u i s a n c e p a r a m e t e r s w h o s e n u m b e r i n c r e a s e s in p r o p o r t i o n to
t h e n u m b e r of o b s e r v a t i o n s . H e r e , t h e H u b e r t b u n d l e t h e o r y p l a y s an e s s e n t i a l
r o l e . C h a p t e r 7 e l u c i d a t e s g e o m e t r i c a l s t r u c t u r e s of p a r a m e t r i c and n o n - p a r a -
m e t r i c m o d e l s
of
s t a t i o n a r y G a u s s i a n t i m e s e r i e s . T h e p r e s e n t a p p r o a c h
is
u s e -
f ul n o t o n l y f o r c o n s t r u c t i n g
a
h i g h e r - o r d e r t h e o r y
of
s t a t i s t i c a l i n f e r e n c e
on
t i m e s e r i e s m o d e l s , b u t a l s o f o r c o n s t r u c t i n g d i f f e r e n t i a l g e o m e t r i c a l t h e o r y of
s y s t e m s and i n f o r m a t i o n t h e o r y ( A m a r i , 1 9 8 3 c ) . T h e s e t h r e e c h a p t e r s are
o r i g i n a l and o n l y s k e t c h e s a r e g i v e n in t h e p r e s e n t p a p e r . M o r e d e t a i l e d t h e o -
r e t i c a l t r e a t m e n t s
and
t h e i r a p p l i c a t i o n s w i l l a p p e a r
as
s e p a r a t e p a p e r s
in the
n e a r f u t u r e .
8/18/2019 Differential Geometry in Statistical Inference
31/246
2 .
G E O M E T R I C A L S T R U C T U R E O F S T A T I S T I C A L M O D E L S
M e t r i c a n d α c o n n e c t i o n
L e t S = { p ( x , θ ) } b e a s t a t i s t i c a l m o d e l c o n s i s t i n g o f p r o b a b i l i t y
d e n s i t y f u n c t i o n s p ( x , θ ) o f r a n d o m v a r i a b l e x ε X w i t h r e s p e c t t o a m e a s u r e P o n
X s u c h t h a t e y e r y d i s t r i b u t i o n i s u n i q u e l y p a r a m e t r i z e d b y a n n d i m e n s i o n a l
v e c t o r p a r a m e t e r θ = ( θ
1
) = ( θ
, . . . , θ
n
) .
S i n c e t h e s e t ί p ( x ) } o f a l l t h e d e n
s i t y f u n c t i o n s o n X i s a s u b s e t o f t h e L , s p a c e o f f u n c t i o n s i n x , S i s c o n s i d
e r e d t o b e a s u b s e t o f t h e L . s p a c e . A s t a t i s t i c a l m o d e l S is s a i d t o b e g e o
m e t r i c a l l y r e g u l a r , w h e n i t s a t i s f i e s t h e f o l l o w i n g r e g u l a r i t y c o n d i t i o n s
A. A g , a n d S i s r e g a r d e d a s a n n d i m e n s i o n a l m a n i f o l d w i t h a c o o r d i n a t e s y s t e m
θ .
A.. T h e d o m a i n Θ o f t h e p a r a m e t e r θ i s h o m e o m o r p h i c t o a n n d i m e n
s i o n a l E u c l i d e a n s p a c e R
n
.
A
2
T h e t o p o l o g y o f S i n d u c e d f r o m R
n
i s c o m p a t i b l e w i t h t h e
r e l a t i v e t o p o l o g y o f S i n t h e L. s p a c e .
A
3
T h e s u p p o r t o f p ( x , θ ) i s c o m m o n f o r a ll θ ε θ , s o t h a t p ( x , θ )
a r e m u t u a l l y a b s o l u t e l y c o n t i n u o u s .
A * . E v e r y d e n s i t y f u n c t i o n p ( x , θ ) is a s m o o t h f u n c t i o n i n Θ
u n i f o r m l y i n x , a n d t h e p a r t i a l d e r i v a t i v e 9 / a e
1
a n d i n t e g r a t i o n o f l o g p ( x , θ )
w i t h r e s p e c t t o t h e m e a s u r e P ( x ) a r e a l w a y s c o m m u t a t i v e .
A ς . T h e m o m e n t s o f t h e s c o r e f u n c t i o n ( a / 3 θ
Ί
) l o g p ( x , θ ) e x i s t u p t o
t h e t h i r d o r d e r a n d a r e s m o o t h i n θ .
A
c
. T h e F i s h e r i n f o r m a t i o n m a t r i x i s p o s i t i v e d e f i n i t e ,
o
C o n d i t i o n 1 i m p l i e s t h a t S i t s e l f i s h o m e o m o r p h i c t o R
n
. I t i s
2 5
8/18/2019 Differential Geometry in Statistical Inference
32/246
26
Shunichi Amari
F i g u r e 1
possible to weaken Condition 1. Howev er, only local properties a r e treated
here
so
that
w e
assume
it for the
sake
o f
simplicity.
In a
later section,
w e
assume o n e more condition whic h guarantees th e validity of Edgeworth expansions.
Let u s denote by 3. = 3/3Θ
1
th e tangent vec tor e. o f the ith
coordinate curve θ
1
(Fig. 1) at point θ. Then, n such tangent vectors e . = 3.,
i = 1,..., n , span th e tangent spac e T at point θ of the manifold S. Any tan
gent vector AεT is a linear comb ination of the basis vectors 3.,
θ i
A = AV ,
where A are the components o f vector A and Einstein's summation c onvention is
assumed throughout th e paper, so that th e summation Σ is automatically taken
for those indices whi ch ap pear twice i n o n e term once a s a subscript and once as
a superscript. T h e tangent spac e T is a linearized version of a small neigh
borhood at θ of S, and an infinitesimal vec tor dθ = d θ
Ί
3. denotes th e vector
connecting tw o neighboring points θ and θ + dθ or two neighbori ng distributions
p(x,θ) a n d p(x , θ + d θ).
Let us introduce a metric in the tangent space T
Λ
. It can be done
u
by defining th e inner product g
i Ί
(θ) = of two basis vectors 3. an d 3.
at θ. To this e n d , w e represent a vector 3εT. by a functio n 3.£(x,θ) in x,
where £(x,θ) = log p(x,θ) and 3̂ (in
3.. A )
is the partial der iva tive 3/3θ
Ί
.
Then, it is natural to define th e inner product by
g.
(θ) =
= E
[3,£ (x,θ)3,£(x, θ)],
(2.1)
• J J 0 1 J
8/18/2019 Differential Geometry in Statistical Inference
33/246
Differential Geometrical Theory of Statistics
27
w h e r e
E
Q
d e n o t e s t h e e x p e c t a t i o n w i t h r e s p e c t
to p ( x , θ ) .
T h i s g . .
is the
σ
I j
F i s h e r i n f o r m a t i o n m a t r i x . T w o v e c t o r s A a n d B a r e o r t h o g o n a l w h e n
< A , B > = < A
1
a
i
, B
J
' a . > =
A
Ί
B
J
' g . .
= 0.
I t
is
s o m e t i m e s n e c e s s a r y
to
c o m p a r e
a
v e c t o r A ε T
of
t h e t a n g e n t
θ
s p a c e
T
A
at
o n e
p o i n t θ
w i t h
a
v e c t o r B ε T , b e l o n g i n g
to
t h e t a n g e n t s p a c e
A
,
D σ σ
a t a n o t h e r
p o i n t
θ ' . T h i s c a n
be
d o n e
by
c o m p a r i n g t h e b a s i s v e c t o r s
a. at T
w i t h t h e b a s i s v e c t o r s a
1
,
at T ..