Date post: | 08-Jan-2018 |
Category: |
Documents |
Upload: | dominick-lambert |
View: | 224 times |
Download: | 0 times |
Stick-breaking Construction for the Indian Buffet Process
Duke University Machine Learning Group
Presented by Kai Ni
July 27, 2007
Yee Whye The, Dilan Gorur, and Zoubin Ghahramani,AISTATS 2007
Outline
Introduction
Indian Buffet Process (IBP)
Stick-breaking construction for IBP
Slice samplers
Results
Introduction Indian Buffet Process (IBP)
A distribution over binary matrices consisting of N rows (objects) and an unbounded number of columns (features);
1/0 in entry (i,k) indicates feature k present/absent from object i.
An example Objects are movies – “Terminator 2”, “Shrek” and “Shanghai
Knights”;
Features are – “action”, “comedy”, “stars Jackie Chan”;
The matrix can be [101; 010; 110].
Relationship to CRP IBP and CRP are both tools for defining nonparametric
Bayesian models with latent variables.
CRP – Each object belongs to only one of infinitely many latent classes.
IBP – Each object can possess potentially any combination of infinitely many latent features.
Previous Gibbs sampler for IBP is based on CRP. In this paper the author derives a stick-breaking representation for the IBP, and develop efficient slice samplers.
Indiant Buffet Process
Let Z be a random binary N x K matrix, and denote entry (I,k) in Z by zik. For each feature k let uk be the prior probability that feature k is present in an object.
Let be the strength parameter of the IBP, the full model is:
If we integrated out uk and taking the limit of K -> infinity, we obtain the IBP in the situation similar to CRP.
Gibbs sampler for IBP
For new features:
Stick-breaking construction for IBP
Derivation
pdf for each u cdf for each u
cdf for u(1)
pdf for u(1)
Relation to DP
Stick-breaking for IBP (2)
In truncated stick-breaking for IBP, let K* be the truncation level. We set u(k)=0 for k>K*, and zik=0 for k>K*.
Slice Sampler
Using Adaptive rejection sampling (ARS) to deal with the truncation level. Introduce an auxiliary slice variable s with
1. Update s: if new s makes K* becomes larger, we iteratively draw u(k) until u(K*’) > s.
2. Update Z: given s, we only need to update zik for each i and k<=K*.
3. Update for k = 1,…, K*.
4. Update u(k) for k = 1, …, K*.
Sampling
k
1 2 K* K*’ Decreasing u(k)
Old s New s
0Range of uniform dist. for s
Change of Representations IBP – ignoring the ordering on features; Stick-breaking IBP – enforcing an ordering with
decreasing weights.
Stick-breaking -> IBP: Drop the stick lengths and the inactive features, leaving only the K+ active feature columns along with the corresponding parameters.
IBP -> stick-breaking: Draw both the stick lengths and order the features in decreasing stick lengths, introducing Ko inactive features until
Semi-ordered Stick-breaking uk
+ on active features are unordered and draw from a CRP similar distribution:
The stick length on inactive feature is similar to the stick-breaking IBP
The auxiliary variable s determines how many inactive features need to add.
(unordered 1~K+) Ko
s0Range of uniform dist. for s
Min(u(k))
Results Used the conjugate linear-Gaussian binary latent
feature model for comparing the performance of the different samplers. Each data point is modeled using a spherical Gaussian with mean zi,:A and variance 2
X
Demonstration
Apply semi-ordered slice sampler to 1000 examples of handwritten images of 3’s in the MNIST dataset.
Conclusion
The author derived novel stick-breaking representations of the Indian buffet process.
Based on these representations, new MCMC samplers are proposed that are easy to implement and work on more general models than Gibbs sampling.