Post on 25-Feb-2016
description
transcript
Mean Field Inference in Dependency Networks: An Empirical Study
Daniel Lowd and Arash ShamaeiUniversity of Oregon
2
Learning and Inference inGraphical Models
We want to learn a probability distribution from data and use it to answer queries.
Applications: medical diagnosis, fault diagnosis, web usage analysis, bioinformatics, collaborative filtering, etc.
A
B C
Answers!Data Model
Learning Inference
3
One-Slide Summary
1. In dependency networks, mean field inference is faster than Gibbs sampling, with similar accuracy.
2. Dependency networks are competitive with Bayesian networks.
A
B C
Answers!Data Model
Learning Inference
4
Outline
• Graphical models:Dependency networks vs. others– Representation– Learning– Inference
• Mean field inference in dependency networks• Experiments
5
Dependency NetworksRepresents a probability distribution over {X1, …, Xn} as a
set of conditional probability distributions.
Example:
€
{P1(X1|X3),P2(X2|X1,X3),P3(X3|X1,X2)}
X1
X2 X3
[Heckerman et al., 2000]
6
Comparison of Graphical Models
Bayesian Network
Markov Network
Dependency Network
Allow cycles? N Y YEasy to learn? Y N YConsistent distribution?
Y Y N
Inferencealgorithms
…lots… …lots… Gibbs,MF (new!)
7
Learning Dependency Networks
For each variable Xi, learn conditional distribution,
€
Pi(X i | X−i)
B=?
<0.2, 0.8>
<0.5, 0.5> <0.7, 0.3>
false
falseC=?
true
true
€
PA (A |B,C) =C=?
<0.7, 0.3> <0.4, 0.6>
falsetrue
€
PB (B |C) =
[Heckerman et al., 2000]
8
Approximate Inference Methods
• Gibbs sampling: Slow but effective• Mean field: Fast and usually accurate• Belief propagation: Fast and usually accurate
A
B C
Answers!Model
9
Gibbs SamplingResample each variable in turn, given its neighbors:
Use set of samples to answer queries. e.g.,
Converges to true distribution, given enough samples (assuming positive distribution).
€
x i(t+1) ~ P X i | X−i = x−i
( t )( )
€
P(B = True) =# of samples where B = True
total # of samples
Previously, the only method used to compute probabilities in DNs.
10
Mean FieldApproximate P with simpler distribution Q:
To find best Q, optimize reverse K-L divergence:
Mean field updates converge to local optimum:
€
P(X1,X2,...,Xn ) ≈Q(X1)Q(X2)L Q(Xn )
€
KL(Q ||P) = Q(X = x)logP(X = x)Q(X = x)x
∑
€
Q(t )(X i) ∝ exp EX − i ~Q ( t−1) (X − i )
logP(X i | X−i)[ ]( )
Works for DNs! Never before tested!
11
Mean Field in Dependency Networks
1. Initialize each Q(Xi) to a uniform distribution.2. Update each Q(Xi) in turn:
3. Stop when marginals Q(Xi) converge.
€
Q(t )(X i) ∝ exp EX − i ~Q ( t−1) (X − i )
logPi(X i | X−i)[ ]( )
If consistent, this is guaranteed to converge.If inconsistent, this always seems to converge in practice.
12
Empirical Questions
Q1. In DNs, how does MF compare to Gibbs sampling in speed and accuracy?
Q2. How do DNs compare to BNs in inference speed and accuracy?
13
Experiments
• Learned DNs and BNs on 12 datasets• Generated queries from test data– Varied evidence variables from 10% to 90%– Score using average CMLL per variable
(conditional marginal log-likelihood):
€
CMLL(x,e) =1
| X |logP(X i = x i | E = e)
i∑
14
Results: Accuracy in DNs
NLTCS
MSNBC
KDDCup 2000Plan
tsAudio
Netflix
Jester
MSWeb
Book
WebKB
Reuter
s-52
20 Newsgr
oups0
0.1
0.2
0.3
0.4
0.5
0.6
DN.MF DN.Gibbs
Neg
ative
CM
LLN
egati
ve C
MLL
15
Results: Timing in DNs (log scale)
NLTCS
MSNBC
KDDCup 2000Plan
tsAudio
Netflix
Jester
MSWeb
Book
WebKB
Reuter
s-52
20 Newsgr
oups0.01
0.1
1
10
100
DN.MF DN.Gibbs
Infe
renc
e Ti
me
(s)
16
MF vs. Gibbs in DNs,run for equal time
Evidence # of MF wins % wins
10% 9 75%
20% 10 83%
30% 10 83%
40% 9 75%
50% 10 83%
60% 10 83%
70% 11 92%
80% 11 92%
90% 12 100%
Average 10.2 85%
In DNs, MF usually more accurate, given equal time.
17
Results: Accuracy
NLTCS
MSNBC
KDDCup 2000Plan
tsAudio
Netflix
Jester
MSWeb
Book
WebKB
Reuter
s-52
20 Newsgr
oups0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
DN.MF DN.Gibbs BN.MF BN.Gibbs BN.BP
Neg
ative
CM
LL
18
Gibbs: DN vs. BNEvidence DN wins Percent wins
10% 3 25%20% 1 8%30% 1 8%40% 3 25%50% 5 42%60% 7 58%70% 10 83%80% 10 83%90% 11 92%
Average 5.7 47%
With more evidence, DNs are more accurate.
19
Experimental Results
Q1. In DNs, how does MF compare to Gibbs sampling in speed and accuracy?
A1. MF is consistently faster with similar accuracy, or more accurate with similar speed.
Q2. How do DNs compare to BNs in inference speed and accuracy?
A2. DNs are competitive with BNs – better with more evidence, worse with less evidence.
20
Conclusion
• MF inference in DNs is fast and accurate, especially with more evidence.
• Future work:– Relational dependency networks
(Neville & Jensen, 2007)
– More powerful approximations
Source code available: http://libra.cs.uoregon.edu/
21
Results: Timing (log scale)
NLTCS
MSNBC
KDDCup 2000Plan
tsAudio
Netflix
Jester
MSWeb
Book
WebKB
Reuter
s-52
20 Newsgr
oups0.01
0.1
1
10
100
1000
DN.MF DN.Gibbs BN.MF BN.Gibbs BN.BP
Infe
renc
e Ti
me
(s)
22
Learned Models
1. Learning time is comparable.2. DNs usually have higher pseudo-likelihood (PLL)3. DNs sometimes have higher log-likelihood (LL)