Statistical Models for Partial Membership

Katherine A. Heller heller@gatsby.ucl.ac.ukSinead Williamson saw56@cam.ac.ukZoubin Ghahramani zoubin@eng.cam.ac.uk

Engineering Department, University of Cambridge, Cambridge, UK

Abstract

We present a principled Bayesian frameworkfor modeling partial memberships of datapoints to clusters. Unlike a standard mix-ture model which assumes that each datapoint belongs to one and only one mixturecomponent, or cluster, a partial membershipmodel allows data points to have fractionalmembership in multiple clusters. Algorithmswhich assign data points partial membershipsto clusters can be useful for tasks such as clus-tering genes based on microarray data (Gasch& Eisen, 2002). Our Bayesian Partial Mem-bership Model (BPM) uses exponential fam-ily distributions to model each cluster, and aproduct of these distibtutions, with weightedparameters, to model each datapoint. Herethe weights correspond to the degree to whichthe datapoint belongs to each cluster. Allparameters in the BPM are continuous, sowe can use Hybrid Monte Carlo to performinference and learning. We discuss relation-ships between the BPM and Latent DirichletAllocation, Mixed Membership models, Ex-ponential Family PCA, and fuzzy clustering.Lastly, we show some experimental resultsand discuss nonparametric extensions to ourmodel.

1. Introduction

The idea of partial membership is quite intuitive andpractically useful. Consider, for example, an individ-ual with a mixed ethnic background, say, partly Asianand partly European. It seems sensible to representthat individual as partly belonging to two differentclasses or sets. Such a partial membership represen-

Appearing in Proceedings of the 25 th International Confer-ence on Machine Learning, Helsinki, Finland, 2008. Copy-right 2008 by the author(s)/owner(s).

tation may be relevant to predicting that individual’sphenotype, or their food preferences. We clearly needmodels that can coherently represent partial member-ship.

Note that partial membership is conceptually very dif-ferent from uncertain membership. Being certain thata person is partly Asian and partly European, is verydifferent than being uncertain about a person’s ethnicbackground. More information about the person, suchas DNA tests, could resolve uncertainty, but cannotmake the person change his ethnic membership.

Partial membership is also the cornerstone of fuzzyset theory. While in traditional set theory, items ei-ther belong to a set or they don’t, fuzzy set theoryequips sets with a membership function µk(x) where0 ≤ µk(x) ≤ 1 denotes the degree to which x partiallybelongs to set k.

In this paper we describe a fully probabilistic approachto data modelling with partial membership. Our ap-proach makes use of a simple way of representing par-tial membership using continuous latent variables. Wedefine a model which can cluster data but which fun-damentally assumes that data points can have par-tial membership in the clusters. Each cluster is repre-sented by an exponential family distribution with con-jugate priors (reviewed in section 3). Our model canbe seen as a continuous latent variable relaxation ofclustering with finite mixture models, and reduces tomixture modelling under certain settings of the hyper-parameters. Unlike Latent Dirichlet Allocation (LDA)(Blei et al., 2003) and Mixed Membership models (Ero-sheva et al., 2004), which also capture partial mem-bership in the form of attribute-specific mixtures, ourmodel does not assume a factorization over attributesand provides a general way of combining exponentialfamily distributions with partial membership. Thecomplete specification of our model is provided in sec-tion 4. Learning and inference are carried out usingMarkov chain Monte Carlo (MCMC) methods. Weshow in particular that because all the parameters in

Statistical Models for Partial Membership

our model are continuous, it is possible to employ a fullhybrid Monte Carlo (HMC) algorithm, which uses gra-dients of the log probability, for inference (section 5).

Our Bayesian Partial Membership (BPM) model bearsinteresting relationships to several well-known mod-els in machine learning and statistics, including LDA(Blei et al., 2003), mixed membership models (Ero-sheva et al., 2004), exponential family PCA (Collinset al., 2002), and Discrete Components Analysis (Bun-tine & Jakulin, 2006). We discuss these relations insection 6, where we also relate our model to fuzzy k-means. In section 7, we present both synthetic andreal-world experimental results using image data andvoting patterns of US senators. We conclude with fu-ture work in section 8.

2. A Partial Membership Model

We can derive our method for modeling partial mem-berships from a standard finite mixture model. In afinite mixture model the probability of a data point,xn given Θ, which contains the parameters for each ofthe K mixture components (clusters) is:

p(xn|Θ) =K∑

k=1

ρkpk(xn|θk) (1)

where pk is the probability distribution of mixturecomponent k, and ρk is the mixing proportion (frac-tion of data points belonging to) for component k1.

Equation 1 can be rewritten using indicator variablesπn = [πn1πn2 . . . πnK ] as follows:

p(xn|Θ) =∑πn

p(πn)K∏

k=1

pk(xn|θk)πnk (2)

where πnk ∈ 0, 1 and∑

k πnk = 1. Here we cannotice that if πnk = 1 this means that data point nbelongs to cluster k (also p(πnk = 1) = ρk). Thereforethe πnk denote memberships of data points to clusters.

In order to obtain a model for partial memberships wecan relax the constraint πnk ∈ 0, 1 to now allow πnk

to take any continuous value in the range [0, 1]. How-ever, in order to compute the probability of the dataunder this continuous relaxation of a finite mixturemodel, we need to modify equation 2 as follows:

p(xn|Θ) =∫

πn

p(πn)1c

K∏k=1

pk(xn|θk)πnkdπn (3)

1This notation differs slightly from standard notationfor mixture models.

Figure 1. Left: A mixture model with two Gaussian mix-ture components, or clusters, can generate data from thetwo distributions shown. Right: Partial membership modelwith the same two clusters can generate data from all thedistributions shown (there are actually infinitely many),which lie between the two original clusters.

The modifications include integrating over all valuesof πn instead of summing, and since the product overclusters K from equation 2 no longer normalizes weput in a normalizing constant c, which is a function ofπn and Θ. Equation 3 now gives us a model for partialmembership.

We illustrate the difference between our partial mem-bership model and a standard mixture model in figure1. Here we can see contours of the Gaussian distri-butions which can generate data in the mixture model(left) and the partial membership model (right), whereboth models are using the same two Gaussian clusters.As an example, if one of these clusters represents theethnicity “White British” and the other cluster repre-sents the ethnicity “Pakistani”, then the figure illus-trates that the partial membership model will be ableto capture someone of mixed ethnicity, whose featuresmay lie in between those of either ethnic group (for ex-ample skin color or nose size), better than the mixturemodel.

3. Conjugate-Exponential Models

In the previous section we derived a partial member-ship model, given by equation 3. However we havenot yet discussed the form of the distribution for eachcluster, pk(xn|θk), and we will now focus on the casewhen these distributions are in the exponential family.

An exponential family distribution can be written inthe form:

pk(xn|θk) = exps(xn)>θk + h(xn) + g(θk) (4)

where s(xn) is a vector depending on the data knownas the sufficient statistics, θk is a vector of natu-ral parameters, h(xn) is a function of the data, andg(θk) is a function of the parameters which ensuresthat the probability normalizes to one when integrat-ing or summing over xn. We will use the short-handxn ∼ Expon(θk) to denote that xn is drawn from an

Statistical Models for Partial Membership

exponential family distribution with natural parame-ters θk.

If we plug the exponential family distribution (equa-tion 4) into our partial membership model (equation3) it follows that:

xn|πn,Θ ∼ Expon(∑

k

πnkθk) (5)

where xn comes from the same exponential family dis-tribution as the original clusters pk, but with new nat-ural parameters which are a convex combination ofthe natural parameters of the original clusters, θk,weighted by πnk, the partial membership values fordata point xn. Computation of the normalizing con-stant c is therefore always tractable when pk is in theexponential family.

A probability distribution p(θk) is said to be conju-gate to the exponential family distribution p(xn|θk) ifp(θk|xn) has the same functional form as p(θk). Inparticular, the conjugate prior to the above exponen-tial family distribution can be written in the form:

p(θ) ∝ expλ>θ + νg(θ) (6)

where λ and ν are hyperparameters of the prior. Wewill use the short-hand, θ ∼ Conj(λ, ν). We now havethe tools to define our Bayesian partial membershipmodel.

4. Bayesian Partial MembershipModels

Consider a model with K clusters, and a data setD = xn : n = 1 . . . N. Let α be a K-dimensionalvector of positive hyperparameters. We start by draw-ing mixture weights from a Dirichlet distribution:

ρ ∼ Dir(α) (7)

Here ρ ∼ Dir(α) is shorthand for p(ρ|α) =c∏K

k=1 ραk−1k where c = Γ(

∑k αk)/

∏k Γ(αk) is a nor-

malization constant which can be expressed in termsof the Gamma function2. For each data point, n, wedraw a partial membership vector πn which representshow much that data point belongs to each of the Kclusters:

πn ∼ Dir(aρ). (8)

The parameter a is a positive scaling constant drawn,for example, from an exponential distribution p(a) =be−ba, where b > 0 is a constant. We assume that

2The Gamma function generalizes the factorial to pos-itive reals: Γ(x) = (x − 1)Γ(x − 1), Γ(n) = (n − 1)! forinteger n

!

!

b

a

!!

!

xN

K

!

Figure 2. Graphical model for the BPM

each cluster k is characterized by an exponential familydistribution with natural parameters θk and that

θk ∼ Conj(λ, ν). (9)

Given all these latent variables, each data point isdrawn from

xn ∼ Expon(∑

k

πnkθk) (10)

In order to get an intuition for what the functions ofthe parameters we have just defined are, we return tothe ethnicity example. Here, each cluster k is an eth-nicity (for example, “White British” and “Pakistani”)and the parameters θk define a distribution over fea-tures for each of the k ethnic groups (for example,how likely it is that someone from that ethnic grouplikes pizza or marmite or bindi bhaji). The parame-ter ρ gives the ethnic composition of the population(for example, 75% “White British” and 25% “Pak-istani”), while a controls how similar to the popu-lation an individual is expected to be (Are 100% ofthe people themselves 75% “White British” and 25%“Pakistani”? Or are 75% of the people 100% “WhiteBritish” and the rest are 100% “Pakistani”? Or some-where in between?). For each person n, πn gives theirindividual ethnic composition, and finally xn givestheir individual feature values (e.g. how much theylike marmite). The graphical model representing thisgenerative process is drawn in Figure 2.

Since the Bayesian Partial Membership Model is a gen-erative model, we tried generating data from it us-ing full-covariance Gaussian clusters. Figure 3 showsthe results of generating 3000 data points from ourmodel with K = 3 clusters as the value of parametera changes. We can see that as the value of a increasesdata points tend to have partial membership in moreclusters. In fact we can prove the following lemmas:

Lemma 1 In the limit that a → 0 the exponentialfamily BPM is a standard mixture model with K com-ponents and mixing proportions ρ.

Statistical Models for Partial Membership

−5 0 5 10−10

−5

0

5

10a = 0.01

−5 0 5 10 15−10

−5

0

5

10a = 0.1

−5 0 5 10 15−10

−5

0

5

10a = 1

−5 0 5 10−10

−5

0

5a = 100

Figure 3. 3000 BPM generated data points with partial as-signments to 3 Gaussian clusters shown in red, as param-eter a varies.

Lemma 2 In the limit that a → ∞ the exponentialfamily BPM model has a single component with naturalparameters

∑k ρkθk.

Proofs of these lemmas follow simply from taking thelimits of equation 8 as a goes to 0 and ∞ respectively.

5. BPM Learning

We can represent the observed data set D as an N×Dmatrix X with rows corresponding to xn, where D isthe number of input features.3 Let Θ be a K × Dmatrix with rows θk and Π be an N ×K matrix withrows πn. Learning in the BPM consists of inferringall unknown variables, Ω = Π,Θ,ρ, a given X. Wetreat the top level variables in the graphical model inFigure 2, Ψ = α,λ, ν, b as fixed hyperparameters,although these could also be learned from data. Ourgoal is to infer p(Ω|X,Ψ), for which we decide to em-ploy Markov chain Monte Carlo (MCMC).

Our key observation for MCMC is that even thoughBPMs contain discrete mixture models as a specialcase, all of the unknown variables Ω of the BPM arecontinuous. Moreover, it is possible to take deriva-tives of the log of the joint probability of all variableswith respect to Ω. This makes it possible to do infer-ence using a full Hybrid Monte Carlo (HMC) algorithmon all parameters. Hybrid (or Hamiltonian) MonteCarlo is an MCMC procedure which overcomes therandom walk behaviour of more traditional Metropo-lis or Gibbs sampling algorithms by making use of thederivatives of the log probability (Neal, 1993; MacKay,

3We assume that the data is represented in its naturalrepresentation for the exponential family likelihood, so thats(xn) = xn.

2003). In high dimensions, this derivative informationcan lead to a dramatically faster mixing of the Markovchain, analogous to how optimization using derivativesis often much faster than using greedy random search.

We start by writing the probability of all parametersand variables4 in our model:

p(X,Ω|Ψ) = p(X|Π,Θ)p(Θ|λ, ν)p(Π|a, ρ)p(a|b)p(ρ|α)(11)

We assume that the hyperparameter ν = 1, and omitit from our derivation. Since the forms of all distri-butions on the right side of equation (11) are given insection 4, we can simply plug these in and see that:

log p(X,Ω|Ψ) =

log Γ(∑

k αk)−∑

k log Γ(αk) +∑

k(αk − 1) log ρk

+ log b− ba + N log Γ (∑

k aρk)−N∑

k log Γ(aρk)+

Pn

Pk(aρk − 1) log πnk +

Pk

ˆθ>k λ + g(θk) + f(λ)

˜+

Pn

ˆ(P

k πnkθk)>xn + h(xn) + g`P

k πnkθk

´˜The Hybrid Monte Carlo algorithm simulates dynam-ics of a system with continuous state Ω on an en-ergy function E(Ω) = − log p(X,Ω|Ψ). The deriva-tives of the energy function ∂E(Ω)

∂Ω) provide forces onthe state variables which encourage the system to findhigh probability regions, while maintaining detailedbalance to ensure that the correct equilibrium distri-bution over states is achieved (Neal, 1993). Since Ωhas constraints, e.g. a > 0 and

∑k ρk = 1, we use a

tranformation of variables so that the new state vari-ables are unconstrained, and we perform dynamics inthis unconstrained space. Specifically, we use a = eη,ρk = erkP

k′ erk′ , and πnk = epnkP

k′ epnk′ . For HMC to be

valid in this new space, the chain rule needs to be ap-plied to the derivatives of E , and the prior needs tobe transformed through the Jacobian of the changeof variables. For example, p(a)da = p(η)dη impliesp(η) = p(a)(da/dη) = ap(a). We also extended theHMC procedure to handle missing inputs in a princi-pled manner, by analytically integrating them out, asthis was required for some of our applications. Moredetails and general pseudocode for HMC can be foundin (MacKay, 2003).

6. Related Work

The BPM model has interesting relations to severalmodels that have been proposed in machine learning,statistics and pattern recognition. We describe theserelationships here.

4A formal distinction between hidden variables, e.g. theπn, and unknown parameters is not necessary as theyare both unknowns.

Statistical Models for Partial Membership

Latent Dirichlet Allocation: Using the notationintroduced above, the BPM model and LDA (Bleiet al., 2003) both incorporate a K-dimensional Dirich-let distributed π variable. In LDA, πn are the mix-ing proportions of the topic mixture for each docu-ment n. Each word in document n can then be seenas having been generated by topic k, with probabilityπnk, where the word distribution for topic k is givenby a multinomial distribution with some parameters,θk. The BPM also combines πnk with some exponen-tial family parameters θk, but here the way in whichthey are combined does not result in a mixture modelfrom which another variable (e.g. a word) is assumedto be generated. In contrast, the data points are in-dexed by n directly, and therefore exist at the doc-ument level of LDA. Each data point is assumed tohave come from an exponential family distribution pa-rameterized by a weighted sum of natural parametersθ, where the weights are given by πn for data pointn. In LDA, data is organized at two levels (e.g. docu-ments and words). More generally, mixed membership(MM) models (Erosheva et al., 2004), or admixturemodels, assume that each data attribute (e.g. words)of the data point (e.g. document) is drawn indepen-dently from a mixture distribution given the member-ship vector for the data point, xnd ∼

∑k πnkP (x|θkd).

LDA and mixed membership models do not averagenatural parameters of exponential family distributionslike the BPM. LDA or MM models could not generatethe continuous densities in figure 3 from full-covarianceGaussians. The analagous generative process for MMmodels is given in figure 4. Since data attributes aredrawn independently, the original clusters (not explic-ity shown) are one dimensional and have means at 0,10 and 20 for both attribute dimensions. We can no-tice from the plot that this model always generates amixture of 9 Gaussians, which is a very different be-havior than the BPM, and clearly not as suitable forthe general modeling of partial memberships. LDAonly makes sense when the objects (e.g. documents)being modelled constitute bags of exchangeable sub-objects (e.g. words). Our model makes no such as-sumption. Moreover, in LDA and MM models thereis a discrete latent variable for every sub-object corre-sponding to which mixture component that sub-objectwas drawn from. This large number of discrete latentvariables makes MCMC sampling in LDA potentiallymuch more expensive than in BPM models.

Exponential Family PCA: Our model bears aninteresting relationship to Exponential Family PCA(Collins et al., 2002). EPCA was originally formu-lated as the solution to an optimization problem basedon Bregman divergences, while our model is a fully

−10 0 10 20 30−5

0

5

10

15

20

25a = 0.01

−10 0 10 20 30−5

0

5

10

15

20

25a = 0.1

−10 0 10 20 30−5

0

5

10

15

20

25a = 1

−10 0 10 20 30−5

0

5

10

15

20

25a = 100

Figure 4. Generative plot for MM model with 3 Gaussianclusters

probabilistic model in which all parameters can be in-tegrated out via MCMC. However, it is possible tothink of EPCA as the likelihood function of a proba-bilistic model, which coupled with a prior on the pa-rameters, would make it possible to do Bayesian in-ference in EPCA and would render it closer to ourmodel. However, our model was entirely motivated bythe idea of partial membership in clusters, which isenforced by forming convex combinations of the nat-ural parameters of exponential family models, whileEPCA is based on linear combinations of the param-eters. Therefore: EPCA does not naturally reduceto clustering, none of the variables can be interpretedas partial memberships, and the coefficients define aplane rather than a convex region in parameter space.

The recent work of Buntine and Jakulin (Buntine &Jakulin, 2006) focusing on the analysis of discrete datais also closely related to the BPM model. The frame-work of (Buntine & Jakulin, 2006) section III B ex-presses a model for discrete data in terms of linearmixtures of dual exponential family parameters whereMAP inference is performed. Section V B also pro-vides insights on differences between using dual andnatural parameters.

Fuzzy Clustering: The notion that probabilisticmodels are unable to handle partial membership hasbeen used to argue that probability is a subtheory ofor different in character from fuzzy logic (Zadeh, 1965;Kosko, 1992). In this paper we described a probabilis-tic model for partial membership which may be of usein the many application domains where fuzzy cluster-ing has been used.

Fuzzy K-means clustering (Bezdek, 1981) itera-tively minimizes the following objective: J =N∑

n=1

K∑k=1

πγnkd2(xn, ck), where γ > 1 is an exponent pa-

rameter, πnk represents the degree of membership of

Statistical Models for Partial Membership

data point n in cluster k (∑

k πnk = 1), and d2(xn, ck)is a measure of squared distance between data pointxn and cluster center ck. By varying γ it is possi-ble to attain different amounts of partial membership,where the limiting case γ = 1 is K-means with nopartial membership. Although the π parameters rep-resent partial membership, none of the variables haveprobabilistic interpretations.

IOMM: Lastly, this work is related to the Infi-nite Overlapping Mixture Model (IOMM) (Heller &Ghahramani, 2007) in which overlapping clustering isperformed, also by taking products of exponential fam-ily distributions, much like products of experts (Hin-ton, 1999). However in the IOMM the membershipsof data points to clusters are restricted to be binary,which means that it can not model partial member-ship.

7. Experiments

We generated a synthetic binary data set from theBPM, and used this to test BPM learning. The syn-thetic data set had 50 data points which each have32 dimensions and can hold partial memberships in3 clusters. We ran our Hybrid Monte Carlo samplerfor 4000 iterations, burning in the first half. In or-der to compare our learned partial membership assign-ments for data points (ΠL) to the true ones (ΠT ) forthis synthetic data set, we compute (U = ΠLΠ>

L ) and(U∗ = ΠT Π>

T ), which basically give the total amountof cluster membership shared between each pair ofdata points, and is invariant to permutations of clus-ter labels. Both of these matrices can be seen in figure5. One can see that the structure of these two ma-trices is quite similar, and that the BPM is learningthe synthetic data reasonably. For a more quantita-tive measure table 5c gives statistics on the number ofpairs of data points whose learned shared membershipdiffers from the true shared membership by more thana given threshold (the range of this statistic is [0,1]).

We also used the BPM to model two “real-world” datasets. The first is senate roll call data from the 107th UScongress (2001-2002) (Jakulin, 2004), and the secondis a data set of images of sunsets and towers.

The senate roll call data is a matrix of 99 senators (onesenator died in 2002 and neither he nor his replacementis included) by 633 votes. It also includes the outcomeof each vote, which is treated as an additional datapoint (like a senator who always voted the actual out-come). The matrix contained binary features for yeaand nay votes, and we used the BPM to cluster thisdata set using K = 2 clusters. There are missing val-

ues in this dataset but this can easily be dealt with inthe HMC log probability calculations by explicitly rep-resenting both 0 and 1 binary values and leaving outmissing values. The results are given in figure 6. Theline in figure 6 represents the amount of membership ofeach senator in one of the clusters (we used the “Demo-crat” cluster, where senators on the far left have partialmemberships very close to 0, and those on the far righthave partial memberships extremely close to 1). Sincethere are two clusters, and the amount of member-ship always sums to 1 across clusters, the figure looksthe same regardless of whether we are looking at the“Democrat” or “Republican” cluster. We can see thatmost Republicans and Democrats are tightly clusteredat the ends of the line (and have partial membershipsvery close to 0 and 1), but that there is a fractionof senators (around 20%) which lies somewhere rea-sonably in between the extreme partial membershipsof 0 or 1. Interesting properties of this figure includethe location of Senator Jeffords who left the Republi-can party in 2001 to become an independent who cau-cused with the Democrats. Also Senator Chafee who isknown as a moderate Republican and who often votedwith the Democrats (for example, he was the only Re-publican to vote against authorizing the use of forcein Iraq), and Senator Miller a conservative Democratwho supported George Bush over John Kerry in the2004 US Presidential elections. Lastly, it is interestingto note the location of the Outcome data point, whichis very much in the middle. This makes sense since the107th congress was split 50-50 (with Republican DickCheney breaking ties), until Senator Jeffords becamean Independent at which point the Democrats had aone seat majority.

We also tried running both fuzzy k-means clusteringand Dirichlet Process Mixture models (DPMs) on thisdata set. While fuzzy k-means found roughly simi-lar rankings of the senators in terms of membership tothe “Democrat” cluster, the exact ranking and, in par-ticular, the amount of partial membership (πn) eachsenator had in the cluster was very sensitive to thefuzzy exponent parameter, which is typically set byhand. Figure 7a plots the amount of membership forthe Outcome data point in black, as well as the mostextreme Republican, Senator Ensign, in red, and themost extreme Democrat, Senator Schumer, in blue, asa function of the fuzzy exponent parameter. We cansee in this plot that as the assignment of the Outcomedata point begins to reach a value even reasonablyclose to 0.5, the most extreme Republican already has20% membership in the “Democrat” cluster. This re-duction in range does not make sense semantically, andpresents a trade-off between finding reasonable values

Statistical Models for Partial Membership

5 10 15 20 25 30 35 40 45 50

5

10

15

20

25

30

35

40

45

505 10 15 20 25 30 35 40 45 50

5

10

15

20

25

30

35

40

45

50

Statistic Percent

|(U − U∗)| ≤ 0.1 60.40

|(U − U∗)| ≤ 0.2 84.28

|(U − U∗)| ≤ 0.3 95.48

|(U − U∗)| ≤ 0.4 99.68

|(U − U∗)| ≤ 0.5 100.00

Figure 5. a) left - matrix U∗ showing the true shared partial memberships for pairs of data points. b) right - matrix Ushowing the learned shared partial memberships. c) Summary statistics for learned U . Reports the percentage of pairsin U whose difference from U∗ in terms of the amount of shared partial memberships is at most the given threshold (0.1- 0.5).

for πn in the middle of the range, versus at the ex-tremes. This kind of sensitivity to parameters doesnot exist in our BPM model, which models both ex-treme and middle range values well.

We tried using a DPM to model this data set wherewe ran the DPM for 1000 iterations of Gibbs sampling,sampling both assignments and concentration parame-ter. The DPM confidently finds 4 clusters: one clusterconsists solely of Democrats, one consists solely of Re-publicans, the third cluster has 9 of the most moderateDemocrats and Republicans plus the ”vote outcome”variable, and the last cluster has just one member,Hollings (D-SC). Figure 7b is a 100x100 matrix show-ing the overlap of cluster assignments for pairs of sen-ators, averaged over 500 samples (there are no changesin relative assignments, the DPM is completely confi-dent). The interpretation of the data provided by theDPM is very different from the BPM model’s. TheDPM does not use uncertainty in cluster membershipto model Senators with intermediate views. Rather, itcreates an entirely new cluster to model these Sena-tors. This makes sense for the data as viewed by theDPM: there is ample data in the roll calls that theseSenators are moderate — it is not the case that there isuncertainty about whether they fall in line with hard-core Democrats or Republicans. This highlights thefact that the responsibilities in a mixture model (suchas the DPM) cannot and should not be interpretedas partial membership, they are representations of un-certainty in full membership. The BPM model, how-ever, explicitly models the partial membership, andcan, for example, represent the fact that a Senatormight be best characterized as moderate (and quan-tify how moderate they are). In order to quantify thiscomparison we calculated the negative log predictiveprobability (in bits) across senators for the BPM andthe DPM (Table 1). We look at a number of differentmeasures: the mean, median, minimum and maximumnumber of bits required to encode a senator’s votes.We also look at the number of bits needed to encodethe “Outcome” in particular. On all of these measures

Mean Median Min Max “Outcome”

BPM 187 168 93 422 224DPM 196 178 112 412 245

Table 1. Comparison between the BPM and a DPM interms of negative log predictive probability (in bits) acrosssenators.

1 1.5 2 2.5 30

0.2

0.4

0.6

0.8

1

exponent

part

ial m

embe

rshi

ps

Figure 7. a) left - fuzzy k-means: plot of the partial mem-bership values for the Outcome data point (in black) andthe most extreme Republican (in red) and Democrat (inblue) as a function of the fuzzy exponent parameter. b)right - DPMs: an ordered 100x100 matrix showing the frac-tion of times each pair of senators was assigned to the samecluster, averaged over 500 Gibbs sampling iterations.

except for maximum, the BPM performs better thanthe DPM, showing that the BPM is a superior modelfor this data set.

Lastly, we used the BPM to model images of sunsetsand towers. The dataset consisted of 329 images ofsunsets or towers, each of which was represented by240 binary simple texture and color features. Partialassignments to K = 2 clusters were learned, and figure8 provides the result. The top row of the figure is thethree images with the most membership in the “sun-set” cluster, the bottom row contains the three imageswith the most membership in the “tower” cluster, andthe middle row shows the 3 images which have closestto 50/50 membership in each cluster (πnk ≈ 0.5). Inthis dataset, as well as all the datasets described inthis section, our HMC sampler was very fast, givingreasonable results within tens of seconds.

Statistical Models for Partial MembershipE

nsig

n (R

−N

V)

Alla

rd (

R−

CO

)G

ram

m (

R−

TX

)H

atch

(R

−U

T)

Alle

n (R

−V

A)

Enz

i (R

−W

Y)

Gra

ssle

y (R

−IA

)V

oino

vich

(R

−O

H)

Bun

ning

(R

−K

Y)

Kyl

(R

−A

Z)

Gre

gg (

R−

NH

)B

row

nbac

k (R

−K

S)

Rob

erts

(R

−K

S)

B

ond

(R−

MO

)T

hom

as (

R−

WY

)T

hurm

ond

(R−

SC

)C

raig

(R

−ID

)B

enne

tt (R

−U

T)

Nic

kles

(R

−O

K)

Lott

(R−

MS

)B

urns

(R

−M

T)

McC

onne

ll (R

−K

Y)

Sm

ith (

R−

NH

)M

urko

wsk

i (R

−A

K)

Fris

t (R

−T

N)

San

toru

m (

R−

PA

)H

elm

s (R

−N

C)

Inho

fe (

R−

OK

)C

rapo

(R

−ID

)T

hom

pson

(R

−T

N)

Ses

sion

s (R

−A

L)

Hag

el (

R−

NE

)Lu

gar

(R−

IN)

Hut

chis

on (

R−

TX

)S

teve

ns (

R−

AK

)D

omen

ici (

R−

NM

)H

utch

inso

n (R

−A

R)

She

lby

(R−

AL)

C

ampb

ell (

R−

CO

)D

eWin

e (R

−O

H)

Coc

hran

(R

−M

S)

War

ner

(R−

VA

)F

itzge

rald

(R

−IL

)S

mith

(R

−O

R)

McC

ain

(R−

AZ

)C

ollin

s (R

−M

E)

Sno

we

(R−

ME

)S

pect

er (

R−

PA

)O

UT

CO

ME

M

iller

(D

−G

A)

Cha

fee

(R−

RI)

N

elso

n (D

−N

E)

Bre

aux

(D−

LA)

Bau

cus

(D−

MT

)H

ollin

gs (

D−

SC

)Li

ncol

n (D

−A

R)

Cle

land

(D

−G

A)

Land

rieu

(D−

LA)

Car

per

(D−

DE

)B

ayh

(D−

IN)

Jeffo

rds

(I−

VT

)T

orric

elli

(D−

NJ)

C

arna

han

(D−

MO

)B

yrd

(D−

WV

)F

eins

tein

(D

−C

A)

Koh

l (D

−W

I)

Bin

gam

an (

D−

NM

)F

eing

old

(D−

WI)

N

elso

n (D

−F

L)

Lieb

erm

an (

D−

CT

)D

asch

le (

D−

SD

)C

onra

d (D

−N

D)

Can

twel

l (D

−W

A)

Ken

nedy

(D

−M

A)

Aka

ka (

D−

HI)

B

iden

(D

−D

E)

Mur

ray

(D−

WA

)B

oxer

(D

−C

A)

John

son

(D−

SD

)D

urbi

n (D

−IL

)H

arki

n (D

−IA

)M

ikul

ski (

D−

MD

)D

ayto

n (D

−M

N)

Levi

n (D

−M

I)

Leah

y (D

−V

T)

Ree

d (D

−R

I)

Cor

zine

(D

−N

J)

Roc

kefe

ller

(D−

WV

)D

odd

(D−

CT

)In

ouye

(D

−H

I)

Clin

ton

(D−

NY

)S

arba

nes

(D−

MD

)D

orga

n (D

−N

D)

Wyd

en (

D−

OR

)E

dwar

ds (

D−

NC

)G

raha

m (

D−

FL)

K

erry

(D

−M

A)

Rei

d (D

−N

V)

Sta

beno

w (

D−

MI)

S

chum

er (

D−

NY

)

Figure 6. Analysis of the partial membership results on the Senate roll call data from 2001-2002. The line shows amountof membership in the “Democrat” cluster with the left of the line being the lowest and the right the highest.

Figure 8. Tower and Sunset images. The top row are theimages found to have largest membership in the “sunset”cluster, the bottom row are images found to have largestmembership in the “tower” cluster, and the middle row arethe images which have the most even membership in bothclusters.

8. Conclusions and Future Work

In summary, we have described a fully probabilisticapproach to data modelling with partial membershipusing continuous latent variables, which can be seen asa relaxation of clustering with finite mixture models.We employed a full Hybrid Monte Carlo algorithm forinference, and our experience with HMC has been verypositive. Despite the general reputation of MCMCmethods for being slow, our model using HMC seemsto discover sensible partial membership structure aftersurprisingly few samples.

In the future we would like to develop a nonparamet-ric version of this model. The most obvious way to tryto generalize this model would be with a Hierarchi-cal Dirichlet Process (Teh et al., 2006). However, thiswould involve averaging over infinitely many poten-tial clusters, which is both computationally infeasible,and also undesirable from the point of view that eachdata point should have non-zero partial membership

in only a few (certainly finite) number of clusters. Amore promising alternative is to use an Indian BuffetProcess (Griffiths & Ghahramani, 2005), where each 1in a row in an IBP sample matrix would represent acluster in which the data point corresponding to thatrow has non-zero partial membership, and then drawthe continuous values for those partial membershipsconditioned on that IBP matrix.

References

Bezdek, J. (1981). Pattern recognition with fuzzy objectivefunction algorithms. Kluwer.

Blei, D., Ng, A., & Jordan, M. (2003). Latent dirichletallocation. JMLR.

Buntine, W., & Jakulin, A. (2006). LNCS, vol. 3940, chap-ter Discrete Component Analysis. Springer.

Collins, M., Dasgupta, S., & Schapire, R. (2002). A gener-alization of principal components analysis to the expo-nential family. NIPS.

Erosheva, E., Fienberg, S., & Lafferty, J. (2004). Mixedmembership models of scientific publications. PNAS.

Gasch, A., & Eisen, M. (2002). Exploring the conditionalcoregulation of yeast gene expression through fuzzy k-means clustering. Genome Biol., 3.

Griffiths, T., & Ghahramani, Z. (2005). Infinite latent fea-ture models and the indian buffet process (Technical Re-port). Gatsby Computational Neuroscience Unit.

Heller, K., & Ghahramani, Z. (2007). A nonparamet-ric bayesian approach to modeling overlapping clusters.AISTATS.

Hinton, G. (1999). Products of experts. ICANN.Jakulin, A. (2004). http://www.ailab.si/aleks/politics/.Kosko, B. (1992). Neural networks and fuzzy systems.

Prentice Hall.MacKay, D. (2003). Information theory, inference, and

learning algorithms. Cambridge University Press.Neal, R. (1993). Probabilistic inference using markov chain

monte carlo methods (Technical Report). University ofToronto.

Teh, Y., Jordan, M., Beal, M., & Blei, D. (2006). Hierar-chical dirichlet processes. JASA, 101.

Zadeh, L. (1965). Fuzzy sets. Info. and Control, 8.