Post on 22-Feb-2016
description
transcript
Markov Chain Sampling Methods for Dirichlet Process Mixture Models
R.M. Neal
Summarized by Joon Shik Kim12.03.15.(Thu)
Computational Models of Intelligence
Abstract • This article reviews Markov chain methods
for sampling from the posterior distribu-tion of a Dirichlet process mixture model and presents two new classes of methods.
• One new approach is to make Metropolis-Hastings updates of the indicators specify-ing which mixture component is associ-ated with each observation, perhaps sup-plemented with a partial form of Gibbs sampling.
Chinese Restaurant Process (1/2)
Chinese Restaurant Process (2/2)
Introduction (1/2)• Modeling a distribution as a mixture of simpler
distribution is useful both as a nonparametric density estimation method and as a way of identi-fying latent classes that can explain the depen-dencies observed between variables.
• Use of Dirichlet process mixture models has be-come computationally feasible with the develop-ment of Markov chain methods for sampling from the posterior distribution of the parameters of the component distribution and/or of the associations of mixture components with observations.
Introduction (2/2)• In this article, I present two new ap-
proaches to Markov chain sampling. • A very simple method for handling non-
conjugate priors is to use Metropolis-Hast-ings updates with the conditional prior as the proposal distribution.
• A variation of this method may sometimes sample more efficiently, particularly when combined with a partial form of Gibbs sampling.
Dirichlet Process Mixture Models (1/5)
• The basic model applies to data y1,…,yn which we regard as part of an indefinite exchangeable sequence, or equivalent, as being independently drawn from some unknown distribu-tion.
Dirichlet Process Mixture Models (2/5)
• We model the distribution from which the yi are drawn as a mixture of dis-tributions of the form F(θ), with the mixing distribution over θ given G. We let the prior for this mixing distri-bution be a Dirichlet process, with concentration parameter α and base distribution G0.
Dirichlet Process Mixture Models (3/5)
Dirichlet Process Mixture Models (4/5)
Dirichlet Process Mixture Models (5/5)
• If we let K go to infinity, the condi-tional probabilities reach the follow-ing limits:
Gibbs Sampling when Conjugate Priors are used (3/4)
Nested CRP
Day 1 Day 2 Day 3