Post on 08-May-2020
transcript
Shun Watanabe
Tokyo University of Agriculture ante Technology
Neyman-Pearson Test and Hoeffding Test
Beyond IID Workshop@SingaporeJuly, 2017
arXiv:1611.08175
(Binary) Hypothesis Testing
We shall discriminate
Zn = (Z1, . . . , Zn) ⇠ Pn Zn = (Z1, . . . , Zn) ⇠ Qnor
Acceptance region: accept P
type I error probability:
type II error probability:
An ✓ Zn
↵n = Pn(Acn)
�n = Qn(An)
Neyman-Pearson Test
Zn is accepted if
1
n
nX
i=1
⇤(Zi) > ⌧⇤(z) := log
P (z)
Q(z)for log-likelihood ratio (LLR)
[Neyman-Pearson 28]
Neyman-Pearson Test
Zn is accepted if
1
n
nX
i=1
⇤(Zi) > ⌧⇤(z) := log
P (z)
Q(z)for log-likelihood ratio (LLR)
Type I and Type II error trade-off:
[Neyman-Pearson 28]
↵n = P
✓1
n
nX
i=1
⇤(Zi) ⌧
◆�n = Q
✓1
n
nX
i=1
⇤(Zi) > ⌧
◆
Neyman-Pearson Test
Zn is accepted if
1
n
nX
i=1
⇤(Zi) > ⌧⇤(z) := log
P (z)
Q(z)for log-likelihood ratio (LLR)
Type I and Type II error trade-off:
Asymptotically
limn!1
↵n = 0for�n·= exp{�nD(PkQ)}
[Neyman-Pearson 28]
↵n = P
✓1
n
nX
i=1
⇤(Zi) ⌧
◆�n = Q
✓1
n
nX
i=1
⇤(Zi) > ⌧
◆
Neyman-Pearson Test
Zn is accepted if
1
n
nX
i=1
⇤(Zi) > ⌧⇤(z) := log
P (z)
Q(z)for log-likelihood ratio (LLR)
Type I and Type II error trade-off:
Asymptotically
limn!1
↵n = 0for
for ↵n·= exp{�nr}�n
·= exp
⇢� n min
P̃ :D(P̃kP )rD(
˜PkQ)
�
�n·= exp{�nD(PkQ)}
[Neyman-Pearson 28]
↵n = P
✓1
n
nX
i=1
⇤(Zi) ⌧
◆�n = Q
✓1
n
nX
i=1
⇤(Zi) > ⌧
◆
Hoeffding Test
Zn is accepted if type satisfies tZn
D(tZnkP ) < r
[Hoeffding 65]
Hoeffding Test
Zn is accepted if type satisfies
Type I and Type II error trade-off:
tZn
D(tZnkP ) < r
[Hoeffding 65]
↵n = P
✓D(tZnkP ) � r
◆�n = Q
✓D(tZnkP ) < r
◆
Hoeffding Test
Zn is accepted if type satisfies
Type I and Type II error trade-off:
Asymptotically
tZn
D(tZnkP ) < r
[Hoeffding 65]
↵n = P
✓D(tZnkP ) � r
◆�n = Q
✓D(tZnkP ) < r
◆
limn!1
↵n = 0for�n·= exp{�nD(PkQ)}
Hoeffding Test
Zn is accepted if type satisfies
Type I and Type II error trade-off:
Asymptotically
tZn
D(tZnkP ) < r
[Hoeffding 65]
↵n = P
✓D(tZnkP ) � r
◆�n = Q
✓D(tZnkP ) < r
◆
limn!1
↵n = 0for
for ↵n·= exp{�nr}�n
·= exp
⇢� n min
P̃ :D(P̃kP )rD(
˜PkQ)
�
�n·= exp{�nD(PkQ)}
Geometrical Interpretation of Two Tests
Neyman-Pearson test
P × × Q
M(τ)
M(⌧) =
(P̃ :
X
z
P̃ (z)⇤(z) = ⌧
)
E =
⇢˜P :
˜P (z) / exp{s⇤(z)}, s 2 R�
[Csiszár, Amari-Nagaoka, etc.]
Geometrical Interpretation of Two Tests
Neyman-Pearson test
P × × Q
M(τ)
M(⌧) =
(P̃ :
X
z
P̃ (z)⇤(z) = ⌧
)
E =
⇢˜P :
˜P (z) / exp{s⇤(z)}, s 2 R�
P1
P2P3
D(P1kP3) = D(P1kP2) +D(P2kP3)
[Csiszár, Amari-Nagaoka, etc.]
Geometrical Interpretation of Two Tests
Neyman-Pearson test
Hoeffding test
P × × Q
M(τ)
M(⌧) =
(P̃ :
X
z
P̃ (z)⇤(z) = ⌧
)
E =
⇢˜P :
˜P (z) / exp{s⇤(z)}, s 2 R�
P1
P2P3
D(P1kP3) = D(P1kP2) +D(P2kP3)
[Csiszár, Amari-Nagaoka, etc.]
P × × Q
⇢P̃ : D(P̃kP ) < r
�
Finite Length PerformanceNeyman-Pearson test/Hoeffding test:
↵n
�n
Second-Order PerformanceNeyman-Pearson test
[Strassen 62]For non-vanishing type I error
limn!1
↵n "
type II exponent is
� log �n = nD(PkQ)�p
nV (PkQ)Q�1(") +O(log n)
V (PkQ) :=
X
x
P (x)
✓log
P (x)
Q(x)
�D(PkQ)
◆2
Q(a) :=
Z 1
a
1p2⇡
et2
2 dt
Second-Order PerformanceHoeffding test
It is known that asymptotically [Wilks 37],
2nD(tZnkP ) ⇠ �2|Z|�1 chi square distribution of degree |Z|� 1
Second-Order PerformanceHoeffding test
For non-vanishing type I error
limn!1
↵n "
type II exponent is
It is known that asymptotically [Wilks 37],
2nD(tZnkP ) ⇠ �2|Z|�1 chi square distribution of degree |Z|� 1
� log �n = nD(PkQ)�qnV (PkQ)Q�1
�2,|Z|�1(") +O(log n)
Q�2,|Z|�1(a) :=
Z 1
a�2|Z|�1(t)dt
Note thatq
Q�1�2,|Z|�1(") > Q�1(")
Summary of Standard Hypothesis Testing
StandardHypothesis
Testing
Hoeffding Test
• LLR• mixture plane
Neyman-Pearson Test
• divergence• divergence sphere
Multiterminal Hypothesis Testing
PXY or QXY
Xn
Y n
f1
f2
g
[Berger 79]M1
M2
Consider zero-rate case: lim
n!1
1
nlog kMik = 0
eg) send marginal types
Multiterminal Hypothesis Testing
PXY or QXY
Xn
Y n
f1
f2
g
[Berger 79]M1
M2
Consider zero-rate case: lim
n!1
1
nlog kMik = 0
eg) send marginal types
Tests for the standard hypothesis testing cannot be used…
Multiterminal Hypothesis Testing
PXY or QXY
Xn
Y n
f1
f2
g
[Berger 79]M1
M2
Consider zero-rate case: lim
n!1
1
nlog kMik = 0
eg) send marginal types
For example, we can consider the following test:
tXn 2 T nX,� and tY n 2 T n
Y,� =) accept PXY
Tests for the standard hypothesis testing cannot be used…
Multiterminal Hypothesis Testing
PXY or QXY
Xn
Y n
f1
f2
g
[Berger 79]M1
M2
Consider zero-rate case: lim
n!1
1
nlog kMik = 0
eg) send marginal types
Proposition [Han 87, Shalaby-Papamarcou 92]
for limn!1
↵n = 0�n·= exp{�nE(PXY kQXY )}
E(PXY kQXY ) := minP̃XY :
P̃X=PX,P̃Y =PY
D(P̃XY kQXY )
Geometrical Interpretation[Amari-Han 89]
PXY
×
× ×
P ∗
XY
QXY
E(P )
E(Q)
M(P )
×
M(Q)
Q∗
XY
M(P ) =
⇢P̃XY : P̃X = PX , P̃Y = PY
�
E(Q) =
⇢˜
QXY : 9 a, b, log
˜
QXY (x, y)
QXY (x, y)= a(x) + b(y)
�
M(Q) =
⇢Q̃XY : Q̃X = QX , Q̃Y = QY
�
E(P ) =
⇢˜
PXY : 9a, b, log
˜
PXY (x, y)
PXY (x, y)= a(x) + b(y)
�
Geometrical Interpretation[Amari-Han 89]
PXY
×
× ×
P ∗
XY
QXY
E(P )
E(Q)
M(P )
×
M(Q)
Q∗
XY
M(P ) =
⇢P̃XY : P̃X = PX , P̃Y = PY
�
E(Q) =
⇢˜
QXY : 9 a, b, log
˜
QXY (x, y)
QXY (x, y)= a(x) + b(y)
�
D(PXY kQXY ) = D(PXY kP ⇤XY ) +D(P ⇤
XY kQXY )
M(Q) =
⇢Q̃XY : Q̃X = QX , Q̃Y = QY
�
= E(PXY kQXY )
E(P ) =
⇢˜
PXY : 9a, b, log
˜
PXY (x, y)
PXY (x, y)= a(x) + b(y)
�
Hoeffding-like Test
is accepted if satisfies[Han-Kobayashi 89]
(tXn , tY n)
E(tXn ⇥ tY nkPXY ) < r
PXY
Hoeffding-like Test
is accepted if satisfies
Type I and Type II error trade-off:
[Han-Kobayashi 89](tXn , tY n)
E(tXn ⇥ tY nkPXY ) < r
Note thatE(tXn ⇥ tY nkPXY ) = E(tXnY nkPXY )
�n = Q
✓E(tXnY nkPXY ) < r
◆↵n = P
✓E(tXnY nkPXY ) � r
◆
PXY
Hoeffding-like Test
is accepted if satisfies
Type I and Type II error trade-off:
Asymptotically
for ↵n·= exp{�nr}
[Han-Kobayashi 89](tXn , tY n)
E(tXn ⇥ tY nkPXY ) < r
Note thatE(tXn ⇥ tY nkPXY ) = E(tXnY nkPXY )
�n·= exp
⇢� n min
E(Q̃XY kQXY )rD(
˜QXY kQXY )
�
which is optimal among zero-rate schemes [Han-Amari 98].
�n = Q
✓E(tXnY nkPXY ) < r
◆↵n = P
✓E(tXnY nkPXY ) � r
◆
PXY
Hoeffding-like Test
PXY
×
××
P ∗
XY
QXY
E(P )
E(Q)
M(P )
⇢Q̃XY : E(Q̃XY kPXY ) < r
�
Neyman-Pearson Test for Multiterminal?
Standard Hypothesis
Testing
MultiterminalHypothesis
Testing
Hoeffding Test
• LLR• mixture plane
Neyman-Pearson Test
• divergence• divergence sphere
Hoeffding-like Test
• projected divergence• divergence cylinder
How to define a proxy of LLRTheorem [SW 16]
For ,
there exists a unique satisfying (P�XY , Q
�XY ) 2 E(P )⇥ E(Q)
�E(QXY kPXY ) < � < E(PXY kQXY )
D(Q�XY kQXY )�D(P�
XY kPXY ) = �
X
x
P
�
XY
(x, y) =X
x
Q
�
XY
(x, y)
X
y
P
�XY (x, y) =
X
y
Q
�XY (x, y)
log
P
�XY (x, y)
PXY (x, y)= c1 log
Q
�XY (x, y)
QXY (x, y)+ c2
for some .c1, c2 2 R\{0}
How to define a proxy of LLR
PXY
×
× ×
P ∗
XY
QXY
E(P )
E(Q)
M(P )
×
M(Q)
Q∗
XY
P̃XY
Q̃XY
(P̃X , P̃Y ) 6= (Q̃X , Q̃Y )
How to define a proxy of LLR
P̃XY
Q̃XY
(P̃X , P̃Y ) 6= (Q̃X , Q̃Y )
PXY
×
× ×
P ∗
XY
QXY
E(P )
E(Q)
M(P )
×
M(Q)
Q∗
XY
How to define a proxy of LLR
PXY
×
× ×
P ∗
XY
QXY
E(P )
E(Q)
M(P )
×
M(Q)
Q∗
XY
P�XY
Q�XY
(P�X , P�
Y ) = (Q�X , Q�
Y )
How to define a proxy of LLR
PXY
×
× ×
P ∗
XY
QXY
E(P )
E(Q)
M(P )
×
M(Q)
Q∗
XY
P�XY
Q�XY
(P�X , P�
Y ) = (Q�X , Q�
Y )
log
P
�XY (x, y)
PXY (x, y)= c1 log
Q
�XY (x, y)
QXY (x, y)+ c2
Parallel condition:
How to define a proxy of LLRFor , define�E(QXY kPXY ) < � < E(PXY kQXY )
⇤�(x, y) := log
Q
�XY (x, y)
QXY (x, y)� log
P
�XY (x, y)
PXY (x, y)
For , define� = E(PXY kQXY )
� = �E(QXY kPXY )
⇤�(x, y) := log
P
⇤XY (x, y)
QXY (x, y)
For , define
⇤�(x, y) := log
PXY (x, y)
Q
⇤XY (x, y)
Neyman-Pearson-like Test
X
x,y
tX
n(x)tY
n(y)⇤�
(x, y) > ⌧
is accepted if satisfies(tXn , tY n)PXY
Neyman-Pearson-like Test
Type I and Type II error trade-off:
Note that implies
X
x,y
tX
n(x)tY
n(y)⇤�
(x, y) > ⌧
(P�XY , Q
�XY ) 2 E(P )⇥ E(Q)
⇤�(x, y) = a(x) + b(y)
for some , which implies a(x), b(x)
X
x,y
tX
n(x)tY
n(y)⇤�
(x, y) =X
x,y
tX
nY
n(x, y)⇤�
(x, y)
↵n = P
✓1
n
nX
i=1
⇤�(Xi, Yi) ⌧
◆�n = Q
✓1
n
nX
i=1
⇤�(Xi, Yi) > ⌧
◆
is accepted if satisfies(tXn , tY n)PXY
Numerical ExamplePXY =
1/2 1/81/8 1/4
�, QXY =
1/8 1/41/2 1/8
�For with , n = 100
Neyman-Pearson-like test outperform Hoeffding-like test:
↵n
�n
Second-Order PerformanceNeyman-Pearson-like test
For limn!1
↵n "
type II exponent is
� log �n = nE(PXY kQXY )�pnV Q�1
(") +O(log n)
Hoeffding-like test
type II exponent is
� log �n = nE(PXY kQXY )�qnV Q�1
�2,k(") +O(log n)
k = (|X |� 1) + (|Y|� 1)
V :=
X
x,y
P
XY
(x, y)
✓log
P
⇤XY
(x, y)
Q
XY
(x, y)
� E(P
XY
kQXY
)
◆2
Large Deviation PerformanceTheorem
Neyman-Pearson-like test with achieve the optimal LDP performance: � = ⌧
for ↵n·= exp{�nr}�n
·= exp
⇢� n min
E(Q̃XY kQXY )rD(
˜QXY kQXY )
�
Adjustment of depending on is very important.� ⌧
� = �E(QXY kPXY )
� = E(PXY kQXY )
� = ⌧ (optimal)
Type I exponent
Type II exponent
Conclusion• There are Neyman-Pearson test and Hoeffding test in the hypothesis testing
Conclusion• There are Neyman-Pearson test and Hoeffding test in the hypothesis testing
P × × Q
M(τ)
Conclusion• There are Neyman-Pearson test and Hoeffding test in the hypothesis testing
P × × Q
M(τ)
P × × Q
Conclusion• There are Neyman-Pearson test and Hoeffding test in the hypothesis testing
P × × Q
M(τ)
P × × Q
PXY
×
××
P ∗
XY
QXY
E(P )
E(Q)
M(P )
Conclusion• There are Neyman-Pearson test and Hoeffding test in the hypothesis testing
P × × Q
M(τ)
P × × Q
PXY
×
××
P ∗
XY
QXY
E(P )
E(Q)
M(P )
PXY
×
× ×
P ∗
XY
QXY
E(P )
E(Q)
M(P )
×
M(Q)
Q∗
XY