A survey on using Bayes reasoning in Data Mining
Directed by : Dr Rahgozar
Mostafa Haghir Chehreghani
Outline
Bayes Theorem MAP, ML hypothesis Minimum description length principle Bayes optimal classifier Naïve Bayes learner summery
Two Roles for Bayesian Methods
Provides practical learning algorithms Naïve Bayes learning Bayesian belief network learning Combine prior knowledge with observed data Requires prior probabilities
Provides useful conceptual framework Provides gold standard for evaluating other
learning algorithms Additional insight into Ockham’s razor
Bayes Theorem
P(h) = prior probability of hypothesis h P(D) = prior probability of training data D P(h|D) = probability of h given D P(D|h) = probability of D given h
)(
)()|()|(
DP
hPhDPDhP
Choosing hypotheses
Generally want the most probable hypothesis given the training dataMaximum a posteriori hypothesis hMAP
If assume P(hi)=P(hj) then can further simplify, and choose the Maximum likelihood (ML) hypothesis
)()|(maxarg
)(
)()|(maxarg
)|(maxarg
hPhDP
DP
hPhDP
DhPh
Hh
Hh
HhMAP
)|(maxarg iHh
ML hDPhi
Why Likelihood Function are Great
MLEs achieve the Cramer-Rao lower bound The CRLB of variance is the inverse of the
derivative of the derivative of the log of the likelihood function. Any estimator β must have a variance greater than or equal to the CRLB
The Neyman-Pearson lemma a likelihood ratio test will have the minimum
possible Type II error of any test with the α that we selected.
Learning a Real Valued Function
Consider any real-valued target function f
Training examples <xi,di>, where di is noisy training value di=f(xi)+ei
ei is random variable (noise) drawn independently for each xi according to some Gaussian distribution with µ=0
Then the maximum likelihood hypothesis hML is the one that minimizes the sum of squared errors
m
iii
HhML xhdh
1
2))((minarg
Learning a Real Valued Function
Maximize natural log of this instead…
m
i
xhd
Hh
m
ii
Hh
HhMAP
ii
e
hdp
hDph
1
)(
2
1
2
1
2
2
1maxarg
)|(maxarg
)|(maxarg
m
iii
Hh
m
iii
Hh
m
i
ii
Hh
m
i
ii
HhML
xhd
xhd
xhd
xhdh
1
2
1
2
1
2
1
2
2
)(minarg
)(maxarg
)(
2
1maxarg
)(
2
1
2
1lnmaxarg
Learning to Predict Probabilities
Consider predicting survival probability from patient data Training examples <xi,di>, where di is 1 or 0
Want to train neural network to output a probability given xi (not a 0 or 1)
In this case can show
Weight update rule for a sigmoid unit:
where
m
iiiii
HhML xhdxhdh
1
))(1ln()1()(lnmaxarg
jkjkjk www
m
iijkiijk xxhdw
1
))((
Minimum Description Length Principle
Ockham’s razor: prefer the shortest hypothesis MDL: prefer the hypothesis h that minimizes
where Lc(x) is the description length of x under encoding C
Example: H=decision trees D=training data labels Lc1(h) is # bits to describe tree h
Lc2(x) is # bits to describe D given h
Hence hMDL trades off tree size for training errors
)|()(minarg21
hDLhLh CCHh
ML
Minimum Description Length Principle
Interesting fact from information theory: The optimal (shortest expected coding length)
code for an event with probability p is –log2p bits So interpret (1):
–log2P(h) is length of h under optimal code –log2P(D|h) is length of D given h under optimal
code
(1) )(log)|(logminarg
)(log)|(logmaxarg
)()|(maxarg
22
22
hPhDP
hPhDP
hPhDPh
Hh
Hh
HhMAP
So far we’ve sought the most probable hypothesis given the data D (ie., hMAP)
Given new instance x what is its most probable classification? hMAP is not the most probable classification!
Consider three possible hypotheses:
Given a new instance x
What’s the most probable classification of x?
3.)|( ,3.)|( ,4.)|( 321 DhPDhPDhP
)( ,)( ,)( 321 xhxhxh
Most Probable Classification of New Instances
Example:
therefore
and
Bayes Optimal Classifier
Hhiij
Vvi
j
DhPhvP )|()|(maxarg
0)|( ,1)|( ,3.)|(
0)|( ,1)|( ,3.)|(
1)|( ,0)|( ,4.)|(
333
222
111
hPhPDhP
hPhPDhP
hPhPDhP
6.)|()|(
4.)|()|(
Hhii
Hhii
i
i
DhPhP
DhPhP
Hhiij
Vvi
j
DhPhvP )|()|(maxarg
Gibbs Classifier
Bayes optimal classier provides best result, but can be expensive if many hypotheses
Gibbs algorithm:1. Choose one hypothesis at random, according to
P(h|D)
2. Use this to classify new instance
Surprising fact: Assume target concepts are drawn at random from H according to priors on H. Then:
][2][ alBayesOptimGibbs errorEerrorE
Naive Bayes Classifier
Along with decision trees, neural networks, nearest nbr, one of the most practical learning methods
When to use Moderate or large training set available Attributes that describe instances are
conditionally independent given classification Successful applications:
Diagnosis Classifying text documents
Naive Bayes Classifier
Assume target function f:X→V, where each instance x described by attributes <a1,a2,…,an>.
Most probable Value of f(x) is:
Naïve Bayes assumption
Which gives
)()|,...,,(maxarg
),...,,(
)()|,...,,(maxarg
),...,,|(maxarg
21
21
21
21
jjnVv
n
jjn
VvMAP
njVv
MAP
vPvaaaP
aaaP
vPvaaaPv
aaavPv
j
j
j
i
jijn vaPvaaaP )|()|,...,,( 21
i
jijVv
NB vaPvPvj
)|()(maxarg
Conditional Independence
Definition: X is conditionally independent of Y given Z if the probability distribution governing X is independent of the value of Y given the value of Z; that is if
more compactly we write
Example: Thunder is conditionally independent of
Rain, given Lightning
)|(),|(),,( kikjikji zZxXPzZyYxXPzyx
)|(),|( ZXPZYXP
Inference in Bayesian Networks
How can one infer the (probabilities of) values of one or more network variables given observed values of others?
Bayes net contains all information needed for this inference
If only one variable with unknown value, easy to infer it In general case, problem is NP hard
In practice, can succeed in many cases Exact inference methods work well for some network
structures Monte Carlo methods simulate the network randomly
to calculate approximate solutions
Learning Bayes Nets
Suppose structure known, variables partially observable
e.g. observe ForestFire, Storm, BusTourGroup,
Thunder, but not Lightning, Campfire,… Similar to training neural network with hidden
units In fact, can learn network conditional
probability tables using gradient ascent! Converge to network h that (locally) maximizes
P(D|h)
More on Learning Bayes Nets
EM algorithm can also be used. Repeatedly:1. Calculate probabilities of unobserved variables,
assuming h
2. Calculate new wijk to maximize E[lnP(D|h)] where D now includes both observed and (calculated probabilities of) unobserved variables
When structure unknown… Algorithms use greedy search to add/substract
edges and nodes Active research topic
Summary
Combine prior knowledge with observed data Impact of prior knowledge (when correct!) is to
lower the sample complexity Active research area
Extend from boolean to real-valued variables Parameterized distributions instead of tables Extend to first-order instead of propositional
systems More effective inference methods …
Reference: Buntine
W. L, (1994). Operations for learning with graphical models. Journal of Artificial Intelligence Research, 2, 159-225.
R. Agrawal, J. Gehrke, D. Gunopolos, and Prabhakar Raghavan. Automatic subspace clustering of high dimensional data for data mining applications. In ACM SIGMOD Conference, 1998.
LEE, C-Y. and ANTONSSON, E.K. 2000. Dynamic partitional clustering using evolution strategies. In Proceedings of the 3rd Asia-Pacific Conference on Simulated Evolution and Learning, Nagoya, Japan.
PELLEG, D. and MOORE, A. 2000. X-means: Extending K-means with Efficient Estimation of the Number of Clusters. In Proceedings 17th ICML, Stanford University.
Reference: Cédric Archambeau, John A. Lee, Michel Verleysen. On
Convergence Problems of the EM Algorithm for Finite Gaussian Mixtures. ESANN'2003 proceedings – European Symposium on Artificial Neural Networks Bruges (Belgium), 23-25 April 2003, d-side publi., ISBN 2-930307-03-X, pp. 99-106
P. Langley and S. Sage. Induction of Selective Bayesian Classifiers. Proc. 10th Conf. on Artificial Intelligence, 1994
J. Bilmes: A Gentle Tutorial of the EM Algorithm and its Application to Parameter stimation for Gaussian Mixture and Hidden Markov Models. Technical Report of the International Computer Science Institute, Berkeley, CA (1998).
Adwait Ratnaparkhi. 1998. Maximum Entropy Models for Natural Language Ambiguity Resolution. Ph.D. thesis, the University of Pennsylvania.
[17] Thorsten Joachims. 1999. Transductive inference for text classification using support vector machines. In Proc. 16th International Conf. on Machine Learning, pages 200–209. Morgan Kaufmann, San Francisco, CA.
Any Question ?