Markov Chain Sampling Methods for Dirichlet Process Mixture Models

R.M. Neal

Summarized by Joon Shik Kim12.03.15.(Thu)

Computational Models of Intelligence

Abstract • This article reviews Markov chain methods

for sampling from the posterior distribu-tion of a Dirichlet process mixture model and presents two new classes of methods.

• One new approach is to make Metropolis-Hastings updates of the indicators specify-ing which mixture component is associ-ated with each observation, perhaps sup-plemented with a partial form of Gibbs sampling.

Chinese Restaurant Process (1/2)

Chinese Restaurant Process (2/2)

Introduction (1/2)• Modeling a distribution as a mixture of simpler

distribution is useful both as a nonparametric density estimation method and as a way of identi-fying latent classes that can explain the depen-dencies observed between variables.

• Use of Dirichlet process mixture models has be-come computationally feasible with the develop-ment of Markov chain methods for sampling from the posterior distribution of the parameters of the component distribution and/or of the associations of mixture components with observations.

Introduction (2/2)• In this article, I present two new ap-

proaches to Markov chain sampling. • A very simple method for handling non-

conjugate priors is to use Metropolis-Hast-ings updates with the conditional prior as the proposal distribution.

• A variation of this method may sometimes sample more efficiently, particularly when combined with a partial form of Gibbs sampling.

Dirichlet Process Mixture Models (1/5)

• The basic model applies to data y1,…,yn which we regard as part of an indefinite exchangeable sequence, or equivalent, as being independently drawn from some unknown distribu-tion.

Dirichlet Process Mixture Models (2/5)

• We model the distribution from which the yi are drawn as a mixture of dis-tributions of the form F(θ), with the mixing distribution over θ given G. We let the prior for this mixing distri-bution be a Dirichlet process, with concentration parameter α and base distribution G0.

Dirichlet Process Mixture Models (3/5)

Dirichlet Process Mixture Models (4/5)

Dirichlet Process Mixture Models (5/5)

• If we let K go to infinity, the condi-tional probabilities reach the follow-ing limits:

Gibbs Sampling when Conjugate Priors are used (3/4)

Nested CRP

Day 1 Day 2 Day 3