・Use  Neural  word  embedding  (e.g.,  word2vec,  Glove)  to  capture  conceptual  similarity  of  words.    à Each  cluster  corresponds  to  one  latent  concept.  

A  Latent  Concept  Topic  Models  for    Robust  Topic  Inference  Using  Word  Embeddings  

MoAvaAon ・Document-­‐level  word  co-­‐occurrence  is  scarce  when  texts  are  short  and  vocabulary  is  diverse  (e.g.  blog,  SNS,  newsgroup).  ・ProbabilisAc  topic  models  (e.g.,  LDA,  pLSI)  infers  topics  based  on  document-­‐level  word  co-­‐occurrence.  à  ConvenFonal  topic  models  are  not  effecFve.  à  Propose  a  novel  topic  model  based  on  co-­‐occurrence  staAsAcs  of  latent  concepts  to  resolve  the  data  sparsity.  

GloVe  vectors  (Pennington+  2014)

Proposal

Gaussian  variance  parameter              controls  the  range  of  the  emission.

Graphical  Models

Latent  Concept  Topic  Model  (LCTM)

Weihua  Hu  (Utokyo)  and  Jun’ichi  Tsujii  (AIST)

Figure 1: Projected latent concepts on the wordembedding space. Concept vectors are annotatedwith their representative concepts in parentheses.

words, we expect topically-related latent conceptsto co-occur many times, even in short texts withdiverse usage of words. This in turn promotestopic inference in LCTM.

LCTM further has the advantage of using con-tinuous word embedding. Traditional LDA as-sumes a fixed vocabulary of word types. Thismodeling assumption prevents LDA from han-dling out of vocabulary (OOV) words in held-outdocuments. On the other hands, since our topicmodel operates on the continuous vector space, itcan naturally handle OOV words once their vectorrepresentation is provided.

The main contributions of our paper are as fol-lows: We propose LCTM that infers topics viadocument-level co-occurrence patterns of latentconcepts, and derive a collapsed Gibbs samplerfor approximate inference. We show that LCTMcan accurately represent short texts by outperform-ing conventional topic models in a clustering task.By means of a classification task, we furthermoredemonstrate that LCTM achieves superior perfor-mance to other state-of-the-art topic models inhandling documents with a high degree of OOVwords.

The remainder of the paper is organized as fol-lows: related work is summarized in Section 2,while LCTM and its inference algorithm are pre-sented in Section 3. Experiments on the 20News-groups are presented in Section 4, and a conclu-sion is presented in Section 5.

2 Related Work

There have been a number of previous studies ontopic models that incorporate word embeddings.The closest model to LCTM is Gaussian LDA

(Das et al., 2015), which models each topic asa Gaussian distribution over the word embeddingspace. However, the assumption that topics areunimodal in the embedding space is not appropri-ate, since topically related words such as ‘neural’and ‘networks’ can occur distantly from each otherin the embedding space. Nguyen et al. (2015) pro-posed topic models that incorporate informationof word vectors in modeling topic-word distribu-tions. Similarly, Petterson et al. (Petterson et al.,2010) exploits external word features to improvethe Dirichlet prior of the topic-word distributions.However, both of the models cannot handle OOVwords, because they assume fixed word types.

Latent concepts in LCTM are closely relatedto ‘constraints’ in interactive topic models (ITM)(Hu et al., 2014). Both latent concepts and con-straints are designed to group conceptually simi-lar words using external knowledge in an attemptto aid topic inference. The difference lies in theirmodeling assumptions: latent concepts in LCTMare modeled as Gaussian distributions over theembedding space, while constraints in ITM aresets of conceptually similar words that are interac-tively identified by humans for each topic. Eachconstraint for each topic is then modeled as amultinomial distribution over the constrained setof words that were identified as mutually relatedby humans. In Section 4, we consider a variant ofITM, whose constraints are instead inferred usingexternal word embeddings.

As regards short texts, a well-known topicmodel is Biterm Topic Model (BTM) (Yan etal., 2013). BTM directly models the genera-tion of biterms (pairs of words) in the whole cor-pus. However, the assumption that pairs of co-occurring words should be assigned to the sametopic might be too strong (Chen et al., 2015).

3 Latent Concept Topic Model

3.1 Generative ModelThe primary difference between LCTM and theconventional topic models is that LCTM describesthe generative process of word vectors in docu-ments, rather than words themselves.

Suppose α and β are parameters for the Dirich-let priors and let vd,i denote the word embeddingfor a word type wd,i. The generative model forLCTM is as follows.

1. For each topic k

(a) Draw a topic concept distribution φk ∼Dirichlet(β).

(a) LDA. (b) LCTM.

Figure 2: Graphical representation.

2. For each latent concept c

(a) Draw a concept vector µc ∼N (µ,σ2

0I).3. For each document d

(a) Draw a document topic distributionθd ∼ Dirichlet(α).

(b) For the i-th word wd,i in document di. Draw its topic assignment zd,i ∼Categorical(θd).

ii. Draw its latent concept assignmentcd,i ∼ Categorical(φzd,i).

iii. Draw a word vector vd,i ∼N (µcd,i ,σ

2I).

The graphical models for LDA and LCTM areshown in Figure 2. Compared to LDA, LCTMadds another layer of latent variables to indicatethe conceptual similarity of words.

3.2 Posterior InferenceIn our application, we observe documents consist-ing of word vectors and wish to infer posterior dis-tributions over all the hidden variables. Since thereis no analytical solution to the posterior, we derivea collapsed Gibbs sampler to perform approximateinference. During the inference, we sample a la-tent concept assignment as well as a topic assign-ment for each word in each document as follows:

p(zd,i = k | cd,i = c,z−d,i, c−d,i,v)

∝(n−d,id,k + αk

)·

n−d,ik,c + βc

n−d,ik,· +

∑c′ βc′

, (1)

P (cd,i = c | zd,i = k,vd,i, z−d,i, c−d,i,v−d,i)

∝(n−d,ik,c + βc

)· N (vd,i|µc,σ

2cI), (2)

where nd,k is the number of words assigned totopic k in document d, and nk,c is the number ofwords assigned to both topic k and latent conceptc. When an index is replaced by ‘·’, the number is

obtained by summing over the index. The super-script −d,i indicates that the current assignmentsof zd,i and cd,i are ignored. N (·|µ,Σ) is a mul-tivariate Gaussian density function with mean µand covariance matrix Σ. µc and σ2

c in Eq. (2)are parameters associated with the latent conceptc and are defined as follows:

µc =1

σ2 + n−d,i·,c σ2

0

⎛

⎝σ2µ+ σ20 ·

∑

(d′,i′)∈A−d,ic

vd′,i′

⎞

⎠ ,

(3)

σ2c =

(1 +

σ20

n−d,i·,c σ2

0 + σ2

)σ2, (4)

where A−d,ic ≡ {(d′, i′) | cd′,i′ = c ∧ (d′, i′) ̸=

(d, i)} (Murphy, 2012). Eq. (1) is similar to thecollapsed Gibbs sampler of LDA (Griffiths andSteyvers, 2004) except that the second term ofEq. (1) is concerned with topic-concept distribu-tions. Eq. (2) of sampling latent concepts has anintuitive interpretation: the first term encouragesconcept assignments that are consistent with thecurrent topic assignment, while the second termencourages concept assignments that are consis-tent with the observed word. The Gaussian vari-ance parameter σ2 acts as a trade-off parameterbetween the two terms via σ2

c . In Section 4.2, westudy the effect of σ2 on document representation.

3.3 Prediction of Topic Proportions

After the posterior inference, the posterior meansof {θd}, {φk} are straightforward to calculate:

θd,k =nd,k + αk

nd,· +∑

k′ αk′, φk,c =

nk,c + βc

nk,· +∑

c′ βc′. (5)

Also posterior means for {µc} are given byEq. (3). We can then use these values to predicta topic proportion θdnew of an unseen documentdnew using collapsed Gibbs sampling as follows:

p(zdnew,i = k | vdnew,i,v−dnew,i,z−dnew,i,φ,µ)

∝(n−dnew,idnew,k + αk

)·∑

c

φk,c

N (vdnew,i|µc,σ2c )∑

c′ N (vdnew,i|µc′ ,σ2c′)

.

(6)

The second term of Eq. (6) is a weighted averageof φk,c with respect to latent concepts. We see thatmore weight is given to the concepts whose corre-sponding vectors µc are closer to the word vec-tor vdnew,i. This to be expected because statisticsof nearby concepts should give more informationabout the word. We also see from Eq. (6) that the

NotaFons  :  Dirichlet  prior  parameters  :  Gaussian  prior  parameters  :  document-­‐topic        distribuFon  :  topic-­‐concept  (word)      distribuFon  :  word  type  :  word  vector  :  latent  topic  :  latent  concept    :  concept  vector  (a) LDA. (b) LCTM.

Figure 2: Graphical representation.

2. For each latent concept c

(a) Draw a concept vector µc ∼N (µ,σ2

0I).3. For each document d

(a) Draw a document topic distributionθd ∼ Dirichlet(α).

(b) For the i-th word wd,i in document di. Draw its topic assignment zd,i ∼Categorical(θd).

ii. Draw its latent concept assignmentcd,i ∼ Categorical(φzd,i).

iii. Draw a word vector vd,i ∼N (µcd,i ,σ

2I).

The graphical models for LDA and LCTM areshown in Figure 2. Compared to LDA, LCTMadds another layer of latent variables to indicatethe conceptual similarity of words.

3.2 Posterior InferenceIn our application, we observe documents consist-ing of word vectors and wish to infer posterior dis-tributions over all the hidden variables. Since thereis no analytical solution to the posterior, we derivea collapsed Gibbs sampler to perform approximateinference. During the inference, we sample a la-tent concept assignment as well as a topic assign-ment for each word in each document as follows:

p(zd,i = k | cd,i = c,z−d,i, c−d,i,v)

∝(n−d,id,k + αk

)·

n−d,ik,c + βc

n−d,ik,· +

∑c′ βc′

, (1)

P (cd,i = c | zd,i = k,vd,i, z−d,i, c−d,i,v−d,i)

∝(n−d,ik,c + βc

)· N (vd,i|µc,σ

2cI), (2)

where nd,k is the number of words assigned totopic k in document d, and nk,c is the number ofwords assigned to both topic k and latent conceptc. When an index is replaced by ‘·’, the number is

obtained by summing over the index. The super-script −d,i indicates that the current assignmentsof zd,i and cd,i are ignored. N (·|µ,Σ) is a mul-tivariate Gaussian density function with mean µand covariance matrix Σ. µc and σ2

c in Eq. (2)are parameters associated with the latent conceptc and are defined as follows:

µc =1

σ2 + n−d,i·,c σ2

0

⎛

⎝σ2µ+ σ20 ·

∑

(d′,i′)∈A−d,ic

vd′,i′

⎞

⎠ ,

(3)

σ2c =

(1 +

σ20

n−d,i·,c σ2

0 + σ2

)σ2, (4)

where A−d,ic ≡ {(d′, i′) | cd′,i′ = c ∧ (d′, i′) ̸=

(d, i)} (Murphy, 2012). Eq. (1) is similar to thecollapsed Gibbs sampler of LDA (Griffiths andSteyvers, 2004) except that the second term ofEq. (1) is concerned with topic-concept distribu-tions. Eq. (2) of sampling latent concepts has anintuitive interpretation: the first term encouragesconcept assignments that are consistent with thecurrent topic assignment, while the second termencourages concept assignments that are consis-tent with the observed word. The Gaussian vari-ance parameter σ2 acts as a trade-off parameterbetween the two terms via σ2

c . In Section 4.2, westudy the effect of σ2 on document representation.

3.3 Prediction of Topic Proportions

After the posterior inference, the posterior meansof {θd}, {φk} are straightforward to calculate:

θd,k =nd,k + αk

nd,· +∑

k′ αk′, φk,c =

nk,c + βc

nk,· +∑

c′ βc′. (5)

Also posterior means for {µc} are given byEq. (3). We can then use these values to predicta topic proportion θdnew of an unseen documentdnew using collapsed Gibbs sampling as follows:

p(zdnew,i = k | vdnew,i,v−dnew,i,z−dnew,i,φ,µ)

∝(n−dnew,idnew,k + αk

)·∑

c

φk,c

N (vdnew,i|µc,σ2c )∑

c′ N (vdnew,i|µc′ ,σ2c′)

.

(6)

The second term of Eq. (6) is a weighted averageof φk,c with respect to latent concepts. We see thatmore weight is given to the concepts whose corre-sponding vectors µc are closer to the word vec-tor vdnew,i. This to be expected because statisticsof nearby concepts should give more informationabout the word. We also see from Eq. (6) that the

(a) LDA. (b) LCTM.

Figure 2: Graphical representation.

2. For each latent concept c

(a) Draw a concept vector µc ∼N (µ,σ2

0I).3. For each document d

(a) Draw a document topic distributionθd ∼ Dirichlet(α).

(b) For the i-th word wd,i in document di. Draw its topic assignment zd,i ∼Categorical(θd).

ii. Draw its latent concept assignmentcd,i ∼ Categorical(φzd,i).

iii. Draw a word vector vd,i ∼N (µcd,i ,σ

2I).

The graphical models for LDA and LCTM areshown in Figure 2. Compared to LDA, LCTMadds another layer of latent variables to indicatethe conceptual similarity of words.

3.2 Posterior InferenceIn our application, we observe documents consist-ing of word vectors and wish to infer posterior dis-tributions over all the hidden variables. Since thereis no analytical solution to the posterior, we derivea collapsed Gibbs sampler to perform approximateinference. During the inference, we sample a la-tent concept assignment as well as a topic assign-ment for each word in each document as follows:

p(zd,i = k | cd,i = c,z−d,i, c−d,i,v)

∝(n−d,id,k + αk

)·

n−d,ik,c + βc

n−d,ik,· +

∑c′ βc′

, (1)

P (cd,i = c | zd,i = k,vd,i, z−d,i, c−d,i,v−d,i)

∝(n−d,ik,c + βc

)· N (vd,i|µc,σ

2cI), (2)

where nd,k is the number of words assigned totopic k in document d, and nk,c is the number ofwords assigned to both topic k and latent conceptc. When an index is replaced by ‘·’, the number is

obtained by summing over the index. The super-script −d,i indicates that the current assignmentsof zd,i and cd,i are ignored. N (·|µ,Σ) is a mul-tivariate Gaussian density function with mean µand covariance matrix Σ. µc and σ2

c in Eq. (2)are parameters associated with the latent conceptc and are defined as follows:

µc =1

σ2 + n−d,i·,c σ2

0

⎛

⎝σ2µ+ σ20 ·

∑

(d′,i′)∈A−d,ic

vd′,i′

⎞

⎠ ,

(3)

σ2c =

(1 +

σ20

n−d,i·,c σ2

0 + σ2

)σ2, (4)

where A−d,ic ≡ {(d′, i′) | cd′,i′ = c ∧ (d′, i′) ̸=

(d, i)} (Murphy, 2012). Eq. (1) is similar to thecollapsed Gibbs sampler of LDA (Griffiths andSteyvers, 2004) except that the second term ofEq. (1) is concerned with topic-concept distribu-tions. Eq. (2) of sampling latent concepts has anintuitive interpretation: the first term encouragesconcept assignments that are consistent with thecurrent topic assignment, while the second termencourages concept assignments that are consis-tent with the observed word. The Gaussian vari-ance parameter σ2 acts as a trade-off parameterbetween the two terms via σ2

c . In Section 4.2, westudy the effect of σ2 on document representation.

3.3 Prediction of Topic Proportions

After the posterior inference, the posterior meansof {θd}, {φk} are straightforward to calculate:

θd,k =nd,k + αk

nd,· +∑

k′ αk′, φk,c =

nk,c + βc

nk,· +∑

c′ βc′. (5)

Also posterior means for {µc} are given byEq. (3). We can then use these values to predicta topic proportion θdnew of an unseen documentdnew using collapsed Gibbs sampling as follows:

p(zdnew,i = k | vdnew,i,v−dnew,i,z−dnew,i,φ,µ)

∝(n−dnew,idnew,k + αk

)·∑

c

φk,c

N (vdnew,i|µc,σ2c )∑

c′ N (vdnew,i|µc′ ,σ2c′)

.

(6)

The second term of Eq. (6) is a weighted averageof φk,c with respect to latent concepts. We see thatmore weight is given to the concepts whose corre-sponding vectors µc are closer to the word vec-tor vdnew,i. This to be expected because statisticsof nearby concepts should give more informationabout the word. We also see from Eq. (6) that the

(a) LDA. (b) LCTM.

Figure 2: Graphical representation.

2. For each latent concept c

(a) Draw a concept vector µc ∼N (µ,σ2

0I).3. For each document d

(a) Draw a document topic distributionθd ∼ Dirichlet(α).

(b) For the i-th word wd,i in document di. Draw its topic assignment zd,i ∼Categorical(θd).

ii. Draw its latent concept assignmentcd,i ∼ Categorical(φzd,i).

iii. Draw a word vector vd,i ∼N (µcd,i ,σ

2I).

The graphical models for LDA and LCTM areshown in Figure 2. Compared to LDA, LCTMadds another layer of latent variables to indicatethe conceptual similarity of words.

3.2 Posterior InferenceIn our application, we observe documents consist-ing of word vectors and wish to infer posterior dis-tributions over all the hidden variables. Since thereis no analytical solution to the posterior, we derivea collapsed Gibbs sampler to perform approximateinference. During the inference, we sample a la-tent concept assignment as well as a topic assign-ment for each word in each document as follows:

p(zd,i = k | cd,i = c,z−d,i, c−d,i,v)

∝(n−d,id,k + αk

)·

n−d,ik,c + βc

n−d,ik,· +

∑c′ βc′

, (1)

P (cd,i = c | zd,i = k,vd,i, z−d,i, c−d,i,v−d,i)

∝(n−d,ik,c + βc

)· N (vd,i|µc,σ

2cI), (2)

where nd,k is the number of words assigned totopic k in document d, and nk,c is the number ofwords assigned to both topic k and latent conceptc. When an index is replaced by ‘·’, the number is

obtained by summing over the index. The super-script −d,i indicates that the current assignmentsof zd,i and cd,i are ignored. N (·|µ,Σ) is a mul-tivariate Gaussian density function with mean µand covariance matrix Σ. µc and σ2

c in Eq. (2)are parameters associated with the latent conceptc and are defined as follows:

µc =1

σ2 + n−d,i·,c σ2

0

⎛

⎝σ2µ+ σ20 ·

∑

(d′,i′)∈A−d,ic

vd′,i′

⎞

⎠ ,

(3)

σ2c =

(1 +

σ20

n−d,i·,c σ2

0 + σ2

)σ2, (4)

where A−d,ic ≡ {(d′, i′) | cd′,i′ = c ∧ (d′, i′) ̸=

(d, i)} (Murphy, 2012). Eq. (1) is similar to thecollapsed Gibbs sampler of LDA (Griffiths andSteyvers, 2004) except that the second term ofEq. (1) is concerned with topic-concept distribu-tions. Eq. (2) of sampling latent concepts has anintuitive interpretation: the first term encouragesconcept assignments that are consistent with thecurrent topic assignment, while the second termencourages concept assignments that are consis-tent with the observed word. The Gaussian vari-ance parameter σ2 acts as a trade-off parameterbetween the two terms via σ2

c . In Section 4.2, westudy the effect of σ2 on document representation.

3.3 Prediction of Topic Proportions

After the posterior inference, the posterior meansof {θd}, {φk} are straightforward to calculate:

θd,k =nd,k + αk

nd,· +∑

k′ αk′, φk,c =

nk,c + βc

nk,· +∑

c′ βc′. (5)

Also posterior means for {µc} are given byEq. (3). We can then use these values to predicta topic proportion θdnew of an unseen documentdnew using collapsed Gibbs sampling as follows:

p(zdnew,i = k | vdnew,i,v−dnew,i,z−dnew,i,φ,µ)

∝(n−dnew,idnew,k + αk

)·∑

c

φk,c

N (vdnew,i|µc,σ2c )∑

c′ N (vdnew,i|µc′ ,σ2c′)

.

(6)

The second term of Eq. (6) is a weighted averageof φk,c with respect to latent concepts. We see thatmore weight is given to the concepts whose corre-sponding vectors µc are closer to the word vec-tor vdnew,i. This to be expected because statisticsof nearby concepts should give more informationabout the word. We also see from Eq. (6) that the

(a) LDA. (b) LCTM.

Figure 2: Graphical representation.

2. For each latent concept c

(a) Draw a concept vector µc ∼N (µ,σ2

0I).3. For each document d

(a) Draw a document topic distributionθd ∼ Dirichlet(α).

(b) For the i-th word wd,i in document di. Draw its topic assignment zd,i ∼Categorical(θd).

ii. Draw its latent concept assignmentcd,i ∼ Categorical(φzd,i).

iii. Draw a word vector vd,i ∼N (µcd,i ,σ

2I).

The graphical models for LDA and LCTM areshown in Figure 2. Compared to LDA, LCTMadds another layer of latent variables to indicatethe conceptual similarity of words.

3.2 Posterior InferenceIn our application, we observe documents consist-ing of word vectors and wish to infer posterior dis-tributions over all the hidden variables. Since thereis no analytical solution to the posterior, we derivea collapsed Gibbs sampler to perform approximateinference. During the inference, we sample a la-tent concept assignment as well as a topic assign-ment for each word in each document as follows:

p(zd,i = k | cd,i = c,z−d,i, c−d,i,v)

∝(n−d,id,k + αk

)·

n−d,ik,c + βc

n−d,ik,· +

∑c′ βc′

, (1)

P (cd,i = c | zd,i = k,vd,i, z−d,i, c−d,i,v−d,i)

∝(n−d,ik,c + βc

)· N (vd,i|µc,σ

2cI), (2)

where nd,k is the number of words assigned totopic k in document d, and nk,c is the number ofwords assigned to both topic k and latent conceptc. When an index is replaced by ‘·’, the number is

obtained by summing over the index. The super-script −d,i indicates that the current assignmentsof zd,i and cd,i are ignored. N (·|µ,Σ) is a mul-tivariate Gaussian density function with mean µand covariance matrix Σ. µc and σ2

c in Eq. (2)are parameters associated with the latent conceptc and are defined as follows:

µc =1

σ2 + n−d,i·,c σ2

0

⎛

⎝σ2µ+ σ20 ·

∑

(d′,i′)∈A−d,ic

vd′,i′

⎞

⎠ ,

(3)

σ2c =

(1 +

σ20

n−d,i·,c σ2

0 + σ2

)σ2, (4)

where A−d,ic ≡ {(d′, i′) | cd′,i′ = c ∧ (d′, i′) ̸=

(d, i)} (Murphy, 2012). Eq. (1) is similar to thecollapsed Gibbs sampler of LDA (Griffiths andSteyvers, 2004) except that the second term ofEq. (1) is concerned with topic-concept distribu-tions. Eq. (2) of sampling latent concepts has anintuitive interpretation: the first term encouragesconcept assignments that are consistent with thecurrent topic assignment, while the second termencourages concept assignments that are consis-tent with the observed word. The Gaussian vari-ance parameter σ2 acts as a trade-off parameterbetween the two terms via σ2

c . In Section 4.2, westudy the effect of σ2 on document representation.

3.3 Prediction of Topic Proportions

After the posterior inference, the posterior meansof {θd}, {φk} are straightforward to calculate:

θd,k =nd,k + αk

nd,· +∑

k′ αk′, φk,c =

nk,c + βc

nk,· +∑

c′ βc′. (5)

Also posterior means for {µc} are given byEq. (3). We can then use these values to predicta topic proportion θdnew of an unseen documentdnew using collapsed Gibbs sampling as follows:

p(zdnew,i = k | vdnew,i,v−dnew,i,z−dnew,i,φ,µ)

∝(n−dnew,idnew,k + αk

)·∑

c

φk,c

N (vdnew,i|µc,σ2c )∑

c′ N (vdnew,i|µc′ ,σ2c′)

.

(6)

The second term of Eq. (6) is a weighted averageof φk,c with respect to latent concepts. We see thatmore weight is given to the concepts whose corre-sponding vectors µc are closer to the word vec-tor vdnew,i. This to be expected because statisticsof nearby concepts should give more informationabout the word. We also see from Eq. (6) that the

(a) LDA. (b) LCTM.

Figure 2: Graphical representation.

2. For each latent concept c

(a) Draw a concept vector µc ∼N (µ,σ2

0I).3. For each document d

(a) Draw a document topic distributionθd ∼ Dirichlet(α).

(b) For the i-th word wd,i in document di. Draw its topic assignment zd,i ∼Categorical(θd).

ii. Draw its latent concept assignmentcd,i ∼ Categorical(φzd,i).

iii. Draw a word vector vd,i ∼N (µcd,i ,σ

2I).

The graphical models for LDA and LCTM areshown in Figure 2. Compared to LDA, LCTMadds another layer of latent variables to indicatethe conceptual similarity of words.

3.2 Posterior InferenceIn our application, we observe documents consist-ing of word vectors and wish to infer posterior dis-tributions over all the hidden variables. Since thereis no analytical solution to the posterior, we derivea collapsed Gibbs sampler to perform approximateinference. During the inference, we sample a la-tent concept assignment as well as a topic assign-ment for each word in each document as follows:

p(zd,i = k | cd,i = c,z−d,i, c−d,i,v)

∝(n−d,id,k + αk

)·

n−d,ik,c + βc

n−d,ik,· +

∑c′ βc′

, (1)

P (cd,i = c | zd,i = k,vd,i, z−d,i, c−d,i,v−d,i)

∝(n−d,ik,c + βc

)· N (vd,i|µc,σ

2cI), (2)

where nd,k is the number of words assigned totopic k in document d, and nk,c is the number ofwords assigned to both topic k and latent conceptc. When an index is replaced by ‘·’, the number is

obtained by summing over the index. The super-script −d,i indicates that the current assignmentsof zd,i and cd,i are ignored. N (·|µ,Σ) is a mul-tivariate Gaussian density function with mean µand covariance matrix Σ. µc and σ2

c in Eq. (2)are parameters associated with the latent conceptc and are defined as follows:

µc =1

σ2 + n−d,i·,c σ2

0

⎛

⎝σ2µ+ σ20 ·

∑

(d′,i′)∈A−d,ic

vd′,i′

⎞

⎠ ,

(3)

σ2c =

(1 +

σ20

n−d,i·,c σ2

0 + σ2

)σ2, (4)

where A−d,ic ≡ {(d′, i′) | cd′,i′ = c ∧ (d′, i′) ̸=

(d, i)} (Murphy, 2012). Eq. (1) is similar to thecollapsed Gibbs sampler of LDA (Griffiths andSteyvers, 2004) except that the second term ofEq. (1) is concerned with topic-concept distribu-tions. Eq. (2) of sampling latent concepts has anintuitive interpretation: the first term encouragesconcept assignments that are consistent with thecurrent topic assignment, while the second termencourages concept assignments that are consis-tent with the observed word. The Gaussian vari-ance parameter σ2 acts as a trade-off parameterbetween the two terms via σ2

c . In Section 4.2, westudy the effect of σ2 on document representation.

3.3 Prediction of Topic Proportions

After the posterior inference, the posterior meansof {θd}, {φk} are straightforward to calculate:

θd,k =nd,k + αk

nd,· +∑

k′ αk′, φk,c =

nk,c + βc

nk,· +∑

c′ βc′. (5)

Also posterior means for {µc} are given byEq. (3). We can then use these values to predicta topic proportion θdnew of an unseen documentdnew using collapsed Gibbs sampling as follows:

p(zdnew,i = k | vdnew,i,v−dnew,i,z−dnew,i,φ,µ)

∝(n−dnew,idnew,k + αk

)·∑

c

φk,c

N (vdnew,i|µc,σ2c )∑

c′ N (vdnew,i|µc′ ,σ2c′)

.

(6)

The second term of Eq. (6) is a weighted averageof φk,c with respect to latent concepts. We see thatmore weight is given to the concepts whose corre-sponding vectors µc are closer to the word vec-tor vdnew,i. This to be expected because statisticsof nearby concepts should give more informationabout the word. We also see from Eq. (6) that the

(a) LDA. (b) LCTM.

Figure 2: Graphical representation.

2. For each latent concept c

(a) Draw a concept vector µc ∼N (µ,σ2

0I).3. For each document d

(a) Draw a document topic distributionθd ∼ Dirichlet(α).

(b) For the i-th word wd,i in document di. Draw its topic assignment zd,i ∼Categorical(θd).

ii. Draw its latent concept assignmentcd,i ∼ Categorical(φzd,i).

iii. Draw a word vector vd,i ∼N (µcd,i ,σ

2I).

The graphical models for LDA and LCTM areshown in Figure 2. Compared to LDA, LCTMadds another layer of latent variables to indicatethe conceptual similarity of words.

3.2 Posterior InferenceIn our application, we observe documents consist-ing of word vectors and wish to infer posterior dis-tributions over all the hidden variables. Since thereis no analytical solution to the posterior, we derivea collapsed Gibbs sampler to perform approximateinference. During the inference, we sample a la-tent concept assignment as well as a topic assign-ment for each word in each document as follows:

p(zd,i = k | cd,i = c,z−d,i, c−d,i,v)

∝(n−d,id,k + αk

)·

n−d,ik,c + βc

n−d,ik,· +

∑c′ βc′

, (1)

P (cd,i = c | zd,i = k,vd,i, z−d,i, c−d,i,v−d,i)

∝(n−d,ik,c + βc

)· N (vd,i|µc,σ

2cI), (2)

where nd,k is the number of words assigned totopic k in document d, and nk,c is the number ofwords assigned to both topic k and latent conceptc. When an index is replaced by ‘·’, the number is

obtained by summing over the index. The super-script −d,i indicates that the current assignmentsof zd,i and cd,i are ignored. N (·|µ,Σ) is a mul-tivariate Gaussian density function with mean µand covariance matrix Σ. µc and σ2

c in Eq. (2)are parameters associated with the latent conceptc and are defined as follows:

µc =1

σ2 + n−d,i·,c σ2

0

⎛

⎝σ2µ+ σ20 ·

∑

(d′,i′)∈A−d,ic

vd′,i′

⎞

⎠ ,

(3)

σ2c =

(1 +

σ20

n−d,i·,c σ2

0 + σ2

)σ2, (4)

where A−d,ic ≡ {(d′, i′) | cd′,i′ = c ∧ (d′, i′) ̸=

(d, i)} (Murphy, 2012). Eq. (1) is similar to thecollapsed Gibbs sampler of LDA (Griffiths andSteyvers, 2004) except that the second term ofEq. (1) is concerned with topic-concept distribu-tions. Eq. (2) of sampling latent concepts has anintuitive interpretation: the first term encouragesconcept assignments that are consistent with thecurrent topic assignment, while the second termencourages concept assignments that are consis-tent with the observed word. The Gaussian vari-ance parameter σ2 acts as a trade-off parameterbetween the two terms via σ2

c . In Section 4.2, westudy the effect of σ2 on document representation.

3.3 Prediction of Topic Proportions

After the posterior inference, the posterior meansof {θd}, {φk} are straightforward to calculate:

θd,k =nd,k + αk

nd,· +∑

k′ αk′, φk,c =

nk,c + βc

nk,· +∑

c′ βc′. (5)

Also posterior means for {µc} are given byEq. (3). We can then use these values to predicta topic proportion θdnew of an unseen documentdnew using collapsed Gibbs sampling as follows:

p(zdnew,i = k | vdnew,i,v−dnew,i,z−dnew,i,φ,µ)

∝(n−dnew,idnew,k + αk

)·∑

c

φk,c

N (vdnew,i|µc,σ2c )∑

c′ N (vdnew,i|µc′ ,σ2c′)

.

(6)

The second term of Eq. (6) is a weighted averageof φk,c with respect to latent concepts. We see thatmore weight is given to the concepts whose corre-sponding vectors µc are closer to the word vec-tor vdnew,i. This to be expected because statisticsof nearby concepts should give more informationabout the word. We also see from Eq. (6) that the

(a) LDA. (b) LCTM.

Figure 2: Graphical representation.

2. For each latent concept c

(a) Draw a concept vector µc ∼N (µ,σ2

0I).3. For each document d

(a) Draw a document topic distributionθd ∼ Dirichlet(α).

(b) For the i-th word wd,i in document di. Draw its topic assignment zd,i ∼Categorical(θd).

ii. Draw its latent concept assignmentcd,i ∼ Categorical(φzd,i).

iii. Draw a word vector vd,i ∼N (µcd,i ,σ

2I).

The graphical models for LDA and LCTM areshown in Figure 2. Compared to LDA, LCTMadds another layer of latent variables to indicatethe conceptual similarity of words.

3.2 Posterior InferenceIn our application, we observe documents consist-ing of word vectors and wish to infer posterior dis-tributions over all the hidden variables. Since thereis no analytical solution to the posterior, we derivea collapsed Gibbs sampler to perform approximateinference. During the inference, we sample a la-tent concept assignment as well as a topic assign-ment for each word in each document as follows:

p(zd,i = k | cd,i = c,z−d,i, c−d,i,v)

∝(n−d,id,k + αk

)·

n−d,ik,c + βc

n−d,ik,· +

∑c′ βc′

, (1)

P (cd,i = c | zd,i = k,vd,i, z−d,i, c−d,i,v−d,i)

∝(n−d,ik,c + βc

)· N (vd,i|µc,σ

2cI), (2)

where nd,k is the number of words assigned totopic k in document d, and nk,c is the number ofwords assigned to both topic k and latent conceptc. When an index is replaced by ‘·’, the number is

obtained by summing over the index. The super-script −d,i indicates that the current assignmentsof zd,i and cd,i are ignored. N (·|µ,Σ) is a mul-tivariate Gaussian density function with mean µand covariance matrix Σ. µc and σ2

c in Eq. (2)are parameters associated with the latent conceptc and are defined as follows:

µc =1

σ2 + n−d,i·,c σ2

0

⎛

⎝σ2µ+ σ20 ·

∑

(d′,i′)∈A−d,ic

vd′,i′

⎞

⎠ ,

(3)

σ2c =

(1 +

σ20

n−d,i·,c σ2

0 + σ2

)σ2, (4)

where A−d,ic ≡ {(d′, i′) | cd′,i′ = c ∧ (d′, i′) ̸=

(d, i)} (Murphy, 2012). Eq. (1) is similar to thecollapsed Gibbs sampler of LDA (Griffiths andSteyvers, 2004) except that the second term ofEq. (1) is concerned with topic-concept distribu-tions. Eq. (2) of sampling latent concepts has anintuitive interpretation: the first term encouragesconcept assignments that are consistent with thecurrent topic assignment, while the second termencourages concept assignments that are consis-tent with the observed word. The Gaussian vari-ance parameter σ2 acts as a trade-off parameterbetween the two terms via σ2

c . In Section 4.2, westudy the effect of σ2 on document representation.

3.3 Prediction of Topic Proportions

After the posterior inference, the posterior meansof {θd}, {φk} are straightforward to calculate:

θd,k =nd,k + αk

nd,· +∑

k′ αk′, φk,c =

nk,c + βc

nk,· +∑

c′ βc′. (5)

Also posterior means for {µc} are given byEq. (3). We can then use these values to predicta topic proportion θdnew of an unseen documentdnew using collapsed Gibbs sampling as follows:

p(zdnew,i = k | vdnew,i,v−dnew,i,z−dnew,i,φ,µ)

∝(n−dnew,idnew,k + αk

)·∑

c

φk,c

N (vdnew,i|µc,σ2c )∑

c′ N (vdnew,i|µc′ ,σ2c′)

.

(6)

The second term of Eq. (6) is a weighted averageof φk,c with respect to latent concepts. We see thatmore weight is given to the concepts whose corre-sponding vectors µc are closer to the word vec-tor vdnew,i. This to be expected because statisticsof nearby concepts should give more informationabout the word. We also see from Eq. (6) that the

(a) LDA. (b) LCTM.

Figure 2: Graphical representation.

2. For each latent concept c

(a) Draw a concept vector µc ∼N (µ,σ2

0I).3. For each document d

(a) Draw a document topic distributionθd ∼ Dirichlet(α).

(b) For the i-th word wd,i in document di. Draw its topic assignment zd,i ∼Categorical(θd).

ii. Draw its latent concept assignmentcd,i ∼ Categorical(φzd,i).

iii. Draw a word vector vd,i ∼N (µcd,i ,σ

2I).

The graphical models for LDA and LCTM areshown in Figure 2. Compared to LDA, LCTMadds another layer of latent variables to indicatethe conceptual similarity of words.

3.2 Posterior InferenceIn our application, we observe documents consist-ing of word vectors and wish to infer posterior dis-tributions over all the hidden variables. Since thereis no analytical solution to the posterior, we derivea collapsed Gibbs sampler to perform approximateinference. During the inference, we sample a la-tent concept assignment as well as a topic assign-ment for each word in each document as follows:

p(zd,i = k | cd,i = c,z−d,i, c−d,i,v)

∝(n−d,id,k + αk

)·

n−d,ik,c + βc

n−d,ik,· +

∑c′ βc′

, (1)

P (cd,i = c | zd,i = k,vd,i, z−d,i, c−d,i,v−d,i)

∝(n−d,ik,c + βc

)· N (vd,i|µc,σ

2cI), (2)

where nd,k is the number of words assigned totopic k in document d, and nk,c is the number ofwords assigned to both topic k and latent conceptc. When an index is replaced by ‘·’, the number is

obtained by summing over the index. The super-script −d,i indicates that the current assignmentsof zd,i and cd,i are ignored. N (·|µ,Σ) is a mul-tivariate Gaussian density function with mean µand covariance matrix Σ. µc and σ2

c in Eq. (2)are parameters associated with the latent conceptc and are defined as follows:

µc =1

σ2 + n−d,i·,c σ2

0

⎛

⎝σ2µ+ σ20 ·

∑

(d′,i′)∈A−d,ic

vd′,i′

⎞

⎠ ,

(3)

σ2c =

(1 +

σ20

n−d,i·,c σ2

0 + σ2

)σ2, (4)

where A−d,ic ≡ {(d′, i′) | cd′,i′ = c ∧ (d′, i′) ̸=

(d, i)} (Murphy, 2012). Eq. (1) is similar to thecollapsed Gibbs sampler of LDA (Griffiths andSteyvers, 2004) except that the second term ofEq. (1) is concerned with topic-concept distribu-tions. Eq. (2) of sampling latent concepts has anintuitive interpretation: the first term encouragesconcept assignments that are consistent with thecurrent topic assignment, while the second termencourages concept assignments that are consis-tent with the observed word. The Gaussian vari-ance parameter σ2 acts as a trade-off parameterbetween the two terms via σ2

c . In Section 4.2, westudy the effect of σ2 on document representation.

3.3 Prediction of Topic Proportions

After the posterior inference, the posterior meansof {θd}, {φk} are straightforward to calculate:

θd,k =nd,k + αk

nd,· +∑

k′ αk′, φk,c =

nk,c + βc

nk,· +∑

c′ βc′. (5)

Also posterior means for {µc} are given byEq. (3). We can then use these values to predicta topic proportion θdnew of an unseen documentdnew using collapsed Gibbs sampling as follows:

p(zdnew,i = k | vdnew,i,v−dnew,i,z−dnew,i,φ,µ)

∝(n−dnew,idnew,k + αk

)·∑

c

φk,c

N (vdnew,i|µc,σ2c )∑

c′ N (vdnew,i|µc′ ,σ2c′)

.

(6)

The second term of Eq. (6) is a weighted averageof φk,c with respect to latent concepts. We see thatmore weight is given to the concepts whose corre-sponding vectors µc are closer to the word vec-tor vdnew,i. This to be expected because statisticsof nearby concepts should give more informationabout the word. We also see from Eq. (6) that the

(a) LDA. (b) LCTM.

Figure 2: Graphical representation.

2. For each latent concept c

(a) Draw a concept vector µc ∼N (µ,σ2

0I).3. For each document d

(a) Draw a document topic distributionθd ∼ Dirichlet(α).

(b) For the i-th word wd,i in document di. Draw its topic assignment zd,i ∼Categorical(θd).

ii. Draw its latent concept assignmentcd,i ∼ Categorical(φzd,i).

iii. Draw a word vector vd,i ∼N (µcd,i ,σ

2I).

The graphical models for LDA and LCTM areshown in Figure 2. Compared to LDA, LCTMadds another layer of latent variables to indicatethe conceptual similarity of words.

3.2 Posterior InferenceIn our application, we observe documents consist-ing of word vectors and wish to infer posterior dis-tributions over all the hidden variables. Since thereis no analytical solution to the posterior, we derivea collapsed Gibbs sampler to perform approximateinference. During the inference, we sample a la-tent concept assignment as well as a topic assign-ment for each word in each document as follows:

p(zd,i = k | cd,i = c,z−d,i, c−d,i,v)

∝(n−d,id,k + αk

)·

n−d,ik,c + βc

n−d,ik,· +

∑c′ βc′

, (1)

P (cd,i = c | zd,i = k,vd,i, z−d,i, c−d,i,v−d,i)

∝(n−d,ik,c + βc

)· N (vd,i|µc,σ

2cI), (2)

where nd,k is the number of words assigned totopic k in document d, and nk,c is the number ofwords assigned to both topic k and latent conceptc. When an index is replaced by ‘·’, the number is

obtained by summing over the index. The super-script −d,i indicates that the current assignmentsof zd,i and cd,i are ignored. N (·|µ,Σ) is a mul-tivariate Gaussian density function with mean µand covariance matrix Σ. µc and σ2

c in Eq. (2)are parameters associated with the latent conceptc and are defined as follows:

µc =1

σ2 + n−d,i·,c σ2

0

⎛

⎝σ2µ+ σ20 ·

∑

(d′,i′)∈A−d,ic

vd′,i′

⎞

⎠ ,

(3)

σ2c =

(1 +

σ20

n−d,i·,c σ2

0 + σ2

)σ2, (4)

where A−d,ic ≡ {(d′, i′) | cd′,i′ = c ∧ (d′, i′) ̸=

(d, i)} (Murphy, 2012). Eq. (1) is similar to thecollapsed Gibbs sampler of LDA (Griffiths andSteyvers, 2004) except that the second term ofEq. (1) is concerned with topic-concept distribu-tions. Eq. (2) of sampling latent concepts has anintuitive interpretation: the first term encouragesconcept assignments that are consistent with thecurrent topic assignment, while the second termencourages concept assignments that are consis-tent with the observed word. The Gaussian vari-ance parameter σ2 acts as a trade-off parameterbetween the two terms via σ2

c . In Section 4.2, westudy the effect of σ2 on document representation.

3.3 Prediction of Topic Proportions

After the posterior inference, the posterior meansof {θd}, {φk} are straightforward to calculate:

θd,k =nd,k + αk

nd,· +∑

k′ αk′, φk,c =

nk,c + βc

nk,· +∑

c′ βc′. (5)

Also posterior means for {µc} are given byEq. (3). We can then use these values to predicta topic proportion θdnew of an unseen documentdnew using collapsed Gibbs sampling as follows:

p(zdnew,i = k | vdnew,i,v−dnew,i,z−dnew,i,φ,µ)

∝(n−dnew,idnew,k + αk

)·∑

c

φk,c

N (vdnew,i|µc,σ2c )∑

c′ N (vdnew,i|µc′ ,σ2c′)

.

(6)

The second term of Eq. (6) is a weighted averageof φk,c with respect to latent concepts. We see thatmore weight is given to the concepts whose corre-sponding vectors µc are closer to the word vec-tor vdnew,i. This to be expected because statisticsof nearby concepts should give more informationabout the word. We also see from Eq. (6) that the

(a) LDA. (b) LCTM.

Figure 2: Graphical representation.

2. For each latent concept c

(a) Draw a concept vector µc ∼N (µ,σ2

0I).3. For each document d

(a) Draw a document topic distributionθd ∼ Dirichlet(α).

(b) For the i-th word wd,i in document di. Draw its topic assignment zd,i ∼Categorical(θd).

ii. Draw its latent concept assignmentcd,i ∼ Categorical(φzd,i).

iii. Draw a word vector vd,i ∼N (µcd,i ,σ

2I).

The graphical models for LDA and LCTM areshown in Figure 2. Compared to LDA, LCTMadds another layer of latent variables to indicatethe conceptual similarity of words.

3.2 Posterior InferenceIn our application, we observe documents consist-ing of word vectors and wish to infer posterior dis-tributions over all the hidden variables. Since thereis no analytical solution to the posterior, we derivea collapsed Gibbs sampler to perform approximateinference. During the inference, we sample a la-tent concept assignment as well as a topic assign-ment for each word in each document as follows:

p(zd,i = k | cd,i = c,z−d,i, c−d,i,v)

∝(n−d,id,k + αk

)·

n−d,ik,c + βc

n−d,ik,· +

∑c′ βc′

, (1)

P (cd,i = c | zd,i = k,vd,i, z−d,i, c−d,i,v−d,i)

∝(n−d,ik,c + βc

)· N (vd,i|µc,σ

2cI), (2)

where nd,k is the number of words assigned totopic k in document d, and nk,c is the number ofwords assigned to both topic k and latent conceptc. When an index is replaced by ‘·’, the number is

obtained by summing over the index. The super-script −d,i indicates that the current assignmentsof zd,i and cd,i are ignored. N (·|µ,Σ) is a mul-tivariate Gaussian density function with mean µand covariance matrix Σ. µc and σ2

c in Eq. (2)are parameters associated with the latent conceptc and are defined as follows:

µc =1

σ2 + n−d,i·,c σ2

0

⎛

⎝σ2µ+ σ20 ·

∑

(d′,i′)∈A−d,ic

vd′,i′

⎞

⎠ ,

(3)

σ2c =

(1 +

σ20

n−d,i·,c σ2

0 + σ2

)σ2, (4)

where A−d,ic ≡ {(d′, i′) | cd′,i′ = c ∧ (d′, i′) ̸=

(d, i)} (Murphy, 2012). Eq. (1) is similar to thecollapsed Gibbs sampler of LDA (Griffiths andSteyvers, 2004) except that the second term ofEq. (1) is concerned with topic-concept distribu-tions. Eq. (2) of sampling latent concepts has anintuitive interpretation: the first term encouragesconcept assignments that are consistent with thecurrent topic assignment, while the second termencourages concept assignments that are consis-tent with the observed word. The Gaussian vari-ance parameter σ2 acts as a trade-off parameterbetween the two terms via σ2

c . In Section 4.2, westudy the effect of σ2 on document representation.

3.3 Prediction of Topic Proportions

After the posterior inference, the posterior meansof {θd}, {φk} are straightforward to calculate:

θd,k =nd,k + αk

nd,· +∑

k′ αk′, φk,c =

nk,c + βc

nk,· +∑

c′ βc′. (5)

Also posterior means for {µc} are given byEq. (3). We can then use these values to predicta topic proportion θdnew of an unseen documentdnew using collapsed Gibbs sampling as follows:

p(zdnew,i = k | vdnew,i,v−dnew,i,z−dnew,i,φ,µ)

∝(n−dnew,idnew,k + αk

)·∑

c

φk,c

N (vdnew,i|µc,σ2c )∑

c′ N (vdnew,i|µc′ ,σ2c′)

.

(6)

The second term of Eq. (6) is a weighted averageof φk,c with respect to latent concepts. We see thatmore weight is given to the concepts whose corre-sponding vectors µc are closer to the word vec-tor vdnew,i. This to be expected because statisticsof nearby concepts should give more informationabout the word. We also see from Eq. (6) that the

Add  another  layer  of  latent  variables  (latent  concepts)  to  mediate  data  sparsity.

(a) LDA. (b) LCTM.

Figure 2: Graphical representation.

2. For each latent concept c

(a) Draw a concept vector µc ∼N (µ,σ2

0I).3. For each document d

(a) Draw a document topic distributionθd ∼ Dirichlet(α).

(b) For the i-th word wd,i in document di. Draw its topic assignment zd,i ∼Categorical(θd).

ii. Draw its latent concept assignmentcd,i ∼ Categorical(φzd,i).

iii. Draw a word vector vd,i ∼N (µcd,i ,σ

2I).

The graphical models for LDA and LCTM areshown in Figure 2. Compared to LDA, LCTMadds another layer of latent variables to indicatethe conceptual similarity of words.

3.2 Posterior InferenceIn our application, we observe documents consist-ing of word vectors and wish to infer posterior dis-tributions over all the hidden variables. Since thereis no analytical solution to the posterior, we derivea collapsed Gibbs sampler to perform approximateinference. During the inference, we sample a la-tent concept assignment as well as a topic assign-ment for each word in each document as follows:

p(zd,i = k | cd,i = c,z−d,i, c−d,i,v)

∝(n−d,id,k + αk

)·

n−d,ik,c + βc

n−d,ik,· +

∑c′ βc′

, (1)

p(cd,i = c | zd,i = k,vd,i, z−d,i, c−d,i,v−d,i)

∝(n−d,ik,c + βc

)· N (vd,i|µc,σ

2cI), (2)

where nd,k is the number of words assigned totopic k in document d, and nk,c is the number ofwords assigned to both topic k and latent conceptc. When an index is replaced by ‘·’, the number is

obtained by summing over the index. The super-script −d,i indicates that the current assignmentsof zd,i and cd,i are ignored. N (·|µ,Σ) is a mul-tivariate Gaussian density function with mean µand covariance matrix Σ. µc and σ2

c in Eq. (2)are parameters associated with the latent conceptc and are defined as follows:

µc =1

σ2 + n−d,i·,c σ2

0

⎛

⎝σ2µ+ σ20 ·

∑

(d′,i′)∈A−d,ic

vd′,i′

⎞

⎠ ,

(3)

σ2c =

(1 +

σ20

n−d,i·,c σ2

0 + σ2

)σ2, (4)

where A−d,ic ≡ {(d′, i′) | cd′,i′ = c ∧ (d′, i′) ̸=

(d, i)} (Murphy, 2012). Eq. (1) is similar to thecollapsed Gibbs sampler of LDA (Griffiths andSteyvers, 2004) except that the second term ofEq. (1) is concerned with topic-concept distribu-tions. Eq. (2) of sampling latent concepts has anintuitive interpretation: the first term encouragesconcept assignments that are consistent with thecurrent topic assignment, while the second termencourages concept assignments that are consis-tent with the observed word. The Gaussian vari-ance parameter σ2 acts as a trade-off parameterbetween the two terms via σ2

c . In Section 4.2, westudy the effect of σ2 on document representation.

3.3 Prediction of Topic Proportions

After the posterior inference, the posterior meansof {θd}, {φk} are straightforward to calculate:

θd,k =nd,k + αk

nd,· +∑

k′ αk′, φk,c =

nk,c + βc

nk,· +∑

c′ βc′. (5)

Also posterior means for {µc} are given byEq. (3). We can then use these values to predicta topic proportion θdnew of an unseen documentdnew using collapsed Gibbs sampling as follows:

p(zdnew,i = k | vdnew,i,v−dnew,i,z−dnew,i,φ,µ)

∝(n−dnew,idnew,k + αk

)·∑

c

φk,c

N (vdnew,i|µc,σ2c )∑

c′ N (vdnew,i|µc′ ,σ2c′)

.

(6)

The second term of Eq. (6) is a weighted averageof φk,c with respect to latent concepts. We see thatmore weight is given to the concepts whose corre-sponding vectors µc are closer to the word vec-tor vdnew,i. This to be expected because statisticsof nearby concepts should give more informationabout the word. We also see from Eq. (6) that the

・Define  topics  as  distribuAons  over  latent  concepts.  à Resolve  data  sparsity  in  short  texts.  ・Model  the  generaAve  process  of  word  embeddings.  à  LCTM  can  naturally  handle  Out  of  Vocabulary  (OOV)  words.    

Overview  of  topic  inference ・Collapsed  Gibbs  sampler  for  the  approximate  inference.    ・Sample  latent  concepts  in  addiFon  to  topics.  

Prop  of  topic  k  in  the  same  doc  

Chapter 1

導入

1.1 概要x ∼ p(x|y = 1)

x ∼ p(x) = p(y = 1)p(x|y = 1) + p(y = −1)p(x|y = −1) (1.1)

= πp(x|y = 1) + (1− π)p(x|y = −1) (1.2)

x ∈ Rd,y ∈ {0, 1}m, s ∈ {0, 1}m: X → Rm

Y = {0, 1}準瞬時 FV符号は，復号遅れを定数シンボルだけ許すことにより，特定の情

報源に対してはハフマン符号より良い圧縮性能を示すことが知られている FV符号である．しかし，一般の情報源に対して，準瞬時 FV符号の圧縮限界がハフマン符号よりも良くなるのかどうかについては知られていない．本論文では，情報源の最頻出シンボルの生起確率が与えられたときの，Binary準瞬時 FV符号のtightな圧縮限界を示すとともに，一般の情報源に対する Binary準瞬時 FV符号の tightな圧縮限界を示す．

1.2 符号

p(zd,i = k | cd,i = c, z−d,i, c−d,i,v) ∝!n−d,id,k + αk

"·

n−d,ik,c + βc

n−d,ik,· +

#c′ βc′

, (1.3)

P (cd,i = c | zd,i = k,vd,i, z−d,i, c−d,i,v−d,i) ∝

!n−d,ik,c + βc

"· N (vd,i|µc,σ

2cI),

(1.4)

D-ary符号とは，情報源シンボル集合 X から有限長の符号語 D∗ への写像として定義される．符号語シンボル集合Dは，{0, 1, . . . , D− 1}として一般性を失

3

Chapter 1

導入

1.1 概要x ∼ p(x|y = 1)

x ∼ p(x) = p(y = 1)p(x|y = 1) + p(y = −1)p(x|y = −1) (1.1)

= πp(x|y = 1) + (1− π)p(x|y = −1) (1.2)

x ∈ Rd,y ∈ {0, 1}m, s ∈ {0, 1}m: X → Rm

Y = {0, 1}準瞬時 FV符号は，復号遅れを定数シンボルだけ許すことにより，特定の情

報源に対してはハフマン符号より良い圧縮性能を示すことが知られている FV符号である．しかし，一般の情報源に対して，準瞬時 FV符号の圧縮限界がハフマン符号よりも良くなるのかどうかについては知られていない．本論文では，情報源の最頻出シンボルの生起確率が与えられたときの，Binary準瞬時 FV符号のtightな圧縮限界を示すとともに，一般の情報源に対する Binary準瞬時 FV符号の tightな圧縮限界を示す．

1.2 符号

p(zd,i = k | cd,i = c, z−d,i, c−d,i,v) ∝!n−d,id,k + αk

"·

n−d,ik,c + βc

n−d,ik,· +

#c′ βc′

, (1.3)

p(cd,i = c | zd,i = k,vd,i, z−d,i, c−d,i,v−d,i) ∝

!n−d,ik,c + βc

"· N (vd,i|µc,σ

2cI),

(1.4)

D-ary符号とは，情報源シンボル集合 X から有限長の符号語 D∗ への写像として定義される．符号語シンボル集合Dは，{0, 1, . . . , D− 1}として一般性を失

3

・Sampling  of  a  topic  assignment

・Sampling  of  a  concept  assignment

Prob  of  topic  k  generaFng  concept  c

Prob  of  topic  k  generaFng  concept  c

Prob  of  concept  c  generaFng  word  vec  v  

:  Gaussian  distribuFon  corresponding  to  latent  concept  c

Chapter 1

導入

1.1 概要x ∼ p(x|y = 1)

x ∼ p(x) = p(y = 1)p(x|y = 1) + p(y = −1)p(x|y = −1) (1.1)

= πp(x|y = 1) + (1− π)p(x|y = −1) (1.2)

x ∈ Rd,y ∈ {0, 1}m, s ∈ {0, 1}m: X → Rm

Y = {0, 1}準瞬時 FV符号は，復号遅れを定数シンボルだけ許すことにより，特定の情

報源に対してはハフマン符号より良い圧縮性能を示すことが知られている FV符号である．しかし，一般の情報源に対して，準瞬時 FV符号の圧縮限界がハフマン符号よりも良くなるのかどうかについては知られていない．本論文では，情報源の最頻出シンボルの生起確率が与えられたときの，Binary準瞬時 FV符号のtightな圧縮限界を示すとともに，一般の情報源に対する Binary準瞬時 FV符号の tightな圧縮限界を示す．

1.2 符号

p(zd,i = k | cd,i = c, z−d,i, c−d,i,v) ∝!n−d,id,k + αk

"·

n−d,ik,c + βc

n−d,ik,· +

#c′ βc′

,

(1.3)

p(cd,i = c | zd,i = k,vd,i, z−d,i, c−d,i,v−d,i) ∝

!n−d,ik,c + βc

"· N (vd,i|µc,σ

2cI),

(1.4)

N (·|µc,σ2cI) (1.5)

3

Experimental  Results

1.  Performance  on  document  clustering  

2.  Performance  on  handling  a  high  degree  of  OOV  words  

Dataset:  Short  posts  (less  than  50  words)  of  20Newsgroup.  

Figure 3: Relationship between σ2 and AMI.

Figure 4: Comparisons on clustering performanceof the topic models.

better handle OOV words in held-out documentsthan LFLDA and nl-cLDA do.

4.3 Representation of Held-out Documentswith OOV words

To show that our model can better predict topicproportions of documents containing OOV wordsthan other topic models, we conducted an exper-iment on a classification task. In particular, weinfer topics from the training dataset and predictedtopic proportions of held-out documents using col-lapsed Gibbs sampler. With the inferred topicproportions on both training dataset and held-outdocuments, we then trained a multi-class classi-fier (multi-class logistic regression implementedin sklearn2 python module) on the training datasetand predicted newsgroup labels of the held-outdocuments.

We compared classification accuracy usingLFLDA, nI-cLDA, LDA, GLDA, LCTM and avariant of LCTM (LCTM-UNK) that ignores OOVin the held-out documents. A higher classifica-tion accuracy indicates a better representation ofunseen documents. Table 2 shows the propor-tion of OOV words and classification accuracyof the held-out documents. We see that LCTM-UNK outperforms other topic models in almost

2See http://scikit-learn.org/stable/.

Training Set 400short 800short 1561shortOOV prop 0.348 0.253 0.181Method Classification AccuracyLCTM 0.302 0.367 0.416LCTM-UNK 0.262 0.340 0.406LFLDA 0.253 0.333 0.410nI-cLDA 0.261 0.333 0.412LDA 0.215 0.293 0.382GLDA 0.0527 0.0529 0.0529Chance Rate 0.0539 0.0539 0.0539

Table 2: Proportions of OOV words and classifi-cation accuracy in the held-out documents.

every setting, demonstrating the superiority ofour method, even when OOV words are ignored.However, the fact that LCTM outperforms LCTM-UNK in all cases clearly illustrates that LCTM caneffectively make use of information about OOV tofurther improve the representation of unseen docu-ments. The results show that the level of improve-ment of LCTM over LCTM-UNK increases as theproportion of OOV becomes greater.

5 Conclusion

In this paper, we have proposed LCTM that iswell suited for application to short texts with di-verse vocabulary. LCTM infers topics accordingto document-level co-occurrence patterns of la-tent concepts, and thus is robust to diverse vocab-ulary usage and data sparsity in short texts. Weshowed experimentally that LCTM can produce asuperior representation of short documents, com-pared to conventional topic models. We addition-ally demonstrated that LCTM can exploit OOV toimprove the representation of unseen documents.Although our paper has focused on improving per-formance of LDA by introducing the latent con-cept for each word, the same idea can be readilyapplied to other topic models that extend LDA.

Acknowledgments

We thank anonymous reviewers for their construc-tive feedback. We also thank Hideki Mima forhelpful discussions and Paul Thompson for in-sightful reviews on the paper. This paper is basedon results obtained from a project commissionedby the New Energy and Industrial Technology De-velopment Organization (NEDO).

Figure 3: Relationship between σ2 and AMI.

Figure 4: Comparisons on clustering performanceof the topic models.

better handle OOV words in held-out documentsthan LFLDA and nl-cLDA do.

4.3 Representation of Held-out Documentswith OOV words

To show that our model can better predict topicproportions of documents containing OOV wordsthan other topic models, we conducted an exper-iment on a classification task. In particular, weinfer topics from the training dataset and predictedtopic proportions of held-out documents using col-lapsed Gibbs sampler. With the inferred topicproportions on both training dataset and held-outdocuments, we then trained a multi-class classi-fier (multi-class logistic regression implementedin sklearn2 python module) on the training datasetand predicted newsgroup labels of the held-outdocuments.

We compared classification accuracy usingLFLDA, nI-cLDA, LDA, GLDA, LCTM and avariant of LCTM (LCTM-UNK) that ignores OOVin the held-out documents. A higher classifica-tion accuracy indicates a better representation ofunseen documents. Table 2 shows the propor-tion of OOV words and classification accuracyof the held-out documents. We see that LCTM-UNK outperforms other topic models in almost

2See http://scikit-learn.org/stable/.

Training Set 400short 800short 1561shortOOV prop 0.348 0.253 0.181Method Classification AccuracyLCTM 0.302 0.367 0.416LCTM-UNK 0.262 0.340 0.406LFLDA 0.253 0.333 0.410nI-cLDA 0.261 0.333 0.412LDA 0.215 0.293 0.382GLDA 0.0527 0.0529 0.0529Chance Rate 0.0539 0.0539 0.0539

Table 2: Proportions of OOV words and classifi-cation accuracy in the held-out documents.

every setting, demonstrating the superiority ofour method, even when OOV words are ignored.However, the fact that LCTM outperforms LCTM-UNK in all cases clearly illustrates that LCTM caneffectively make use of information about OOV tofurther improve the representation of unseen docu-ments. The results show that the level of improve-ment of LCTM over LCTM-UNK increases as theproportion of OOV becomes greater.

5 Conclusion

In this paper, we have proposed LCTM that iswell suited for application to short texts with di-verse vocabulary. LCTM infers topics accordingto document-level co-occurrence patterns of la-tent concepts, and thus is robust to diverse vocab-ulary usage and data sparsity in short texts. Weshowed experimentally that LCTM can produce asuperior representation of short documents, com-pared to conventional topic models. We addition-ally demonstrated that LCTM can exploit OOV toimprove the representation of unseen documents.Although our paper has focused on improving per-formance of LDA by introducing the latent con-cept for each word, the same idea can be readilyapplied to other topic models that extend LDA.

Acknowledgments

We thank anonymous reviewers for their construc-tive feedback. We also thank Hideki Mima forhelpful discussions and Paul Thompson for in-sightful reviews on the paper. This paper is basedon results obtained from a project commissionedby the New Energy and Industrial Technology De-velopment Organization (NEDO).

・Gaussian  variance  with                    consistently  performs  well.  ・LCTM  outperforms  TM  w/o  word  embeddings.  ・LCTM  performs  comparable  to  TM  w/  word  embeddings.  

topic assignment of a word is determined by itsembedding, instead of its word type. Therefore,LCTM can naturally handle OOV words once theirembeddings are provided.

3.4 Reducing the Computational ComplexityFrom Eqs. (1) and (2), we see that the computa-tional complexity of sampling per word is O(K +SD), where K, S and D are numbers of topics, la-tent concepts and embedding dimensions, respec-tively. Since K ≪ S holds in usual settings, thedominant computation involves the sampling oflatent concept, which costs O(SD) computationper word.

However, since LCTM assumes that Gaussianvariance σ2 is relatively small, the chance of aword being assigned to distant concepts is negli-gible. Thus, we can reasonably assume that eachword is assigned to one of M ≪ S nearest con-cepts. Hence, the computational complexity isreduced to O(MD). Since concept vectors canmove slightly in the embedding space during theinference, we periodically update the nearest con-cepts for each word type.

To further reduce the computational complexity,we can apply dimensional reduction algorithmssuch as PCA and t-SNE (Van der Maaten and Hin-ton, 2008) to word embeddings to make D smaller.We leave this to future work.

4 Experiments

4.1 Datasets and Models DescriptionIn this section, we study the empirical perfor-mance of LCTM on short texts. We used the20Newsgroups corpus, which consists of discus-sion posts about various news subjects authoredby diverse readers. Each document in the corpus istagged with one of twenty newsgroups. Only postswith less than 50 words are extracted for trainingdatasets. For external word embeddings, we used50-dimensional GloVe1 that were pre-trained onWikipedia. The datasets are summarized in Ta-ble 1. See appendix A for the detail of the datasetpreprocessing.

We compare the performance of the LCTM tothe following six baselines:

• LFLDA (Nguyen et al., 2015), an extensionof Latent Dirichlet Allocation that incorpo-rates word embeddings information.

1Downloaded athttp://nlp.stanford.edu/projects/glove/

Dataset Doc size Vocab size Avg len400short 400 4729 31.87800short 800 7329 31.781561short 1561 10644 31.83held-out 7235 37944 140.15

Table 1: Summary of datasets.

• LFDMM (Nguyen et al., 2015), an extensionof Dirichlet Multinomial Mixtures that incor-porates word embeddings information.

• nI-cLDA, non-interactive constrained LatentDirichlet Allocatoin, a variant of ITM (Hu etal., 2014), where constraints are inferred byapplying k-means to external word embed-dings. Each resulting word cluster is then re-garded as a constraint. See appendix B forthe detail of the model.

• GLDA (Das et al., 2015), Gaussian LDA.

• BTM (Yan et al., 2013), Biterm Topic Model.

• LDA (Blei et al., 2003).

In all the models, we set the number of topicsto be 20. For LCTM (resp. nI-ITM), we set thenumber of latent concepts (resp. constraints) tobe 1000. See appendix C for the detail of hyper-parameter settings.

4.2 Document ClusteringTo demonstrate that LCTM results in a superiorrepresentation of short documents compared to thebaselines, we evaluated the performance of eachmodel on a document clustering task. We useda learned topic proportion as a feature for eachdocument and applied k-means to cluster the doc-uments. We then compared the resulting clus-ters to the actual newsgroup labels. Clusteringperformance is measured by Adjusted Mutual In-formation (AMI) (Manning et al., 2008). HigherAMI indicates better clustering performance. Fig-ure 3 illustrates the quality of clustering in termsof Gaussian variance parameter σ2. We see thatsetting σ2 = 0.5 consistently obtains good clus-tering performance for all the datasets with vary-ing sizes. We therefore set σ2 = 0.5 in the laterevaluation. Figure 4 compares AMI on four topicmodels. We see that LCTM outperforms the topicmodels without word embeddings. Also, we seethat LCTM performs comparable to LFLDA andnl-cLDA, both of which incorporate informationof word embeddings to aid topic inference. How-ever, as we will see in the next section, LCTM can

Figure 3: Relationship between σ2 and AMI.

Figure 4: Comparisons on clustering performanceof the topic models.

better handle OOV words in held-out documentsthan LFLDA and nl-cLDA do.

4.3 Representation of Held-out Documentswith OOV words

To show that our model can better predict topicproportions of documents containing OOV wordsthan other topic models, we conducted an exper-iment on a classification task. In particular, weinfer topics from the training dataset and predictedtopic proportions of held-out documents using col-lapsed Gibbs sampler. With the inferred topicproportions on both training dataset and held-outdocuments, we then trained a multi-class classi-fier (multi-class logistic regression implementedin sklearn2 python module) on the training datasetand predicted newsgroup labels of the held-outdocuments.

We compared classification accuracy usingLFLDA, nI-cLDA, LDA, GLDA, LCTM and avariant of LCTM (LCTM-UNK) that ignores OOVin the held-out documents. A higher classifica-tion accuracy indicates a better representation ofunseen documents. Table 2 shows the propor-tion of OOV words and classification accuracyof the held-out documents. We see that LCTM-UNK outperforms other topic models in almost

2See http://scikit-learn.org/stable/.

Training Set 400short 800short 1561shortOOV prop 0.348 0.253 0.181Method Classification AccuracyLCTM 0.302 0.367 0.416LCTM-UNK 0.262 0.340 0.406LFLDA 0.253 0.333 0.410nI-cLDA 0.261 0.333 0.412LDA 0.215 0.293 0.382GLDA 0.0527 0.0529 0.0529Chance Rate 0.0539 0.0539 0.0539

Table 2: Proportions of OOV words and classifi-cation accuracy in the held-out documents.

every setting, demonstrating the superiority ofour method, even when OOV words are ignored.However, the fact that LCTM outperforms LCTM-UNK in all cases clearly illustrates that LCTM caneffectively make use of information about OOV tofurther improve the representation of unseen docu-ments. The results show that the level of improve-ment of LCTM over LCTM-UNK increases as theproportion of OOV becomes greater.

5 Conclusion

In this paper, we have proposed LCTM that iswell suited for application to short texts with di-verse vocabulary. LCTM infers topics accordingto document-level co-occurrence patterns of la-tent concepts, and thus is robust to diverse vocab-ulary usage and data sparsity in short texts. Weshowed experimentally that LCTM can produce asuperior representation of short documents, com-pared to conventional topic models. We addition-ally demonstrated that LCTM can exploit OOV toimprove the representation of unseen documents.Although our paper has focused on improving per-formance of LDA by introducing the latent con-cept for each word, the same idea can be readilyapplied to other topic models that extend LDA.

Acknowledgments

We thank anonymous reviewers for their construc-tive feedback. We also thank Hideki Mima forhelpful discussions and Paul Thompson for in-sightful reviews on the paper. This paper is basedon results obtained from a project commissionedby the New Energy and Industrial Technology De-velopment Organization (NEDO).

ClassificaFon  accuracy  on  held-­‐out  documents

Clustering  performance  measured  by  Adjusted  Mutual  InformaFon  (AMI)

・LCTM-­‐UNK  (LCTM  that  ignores  OOV)  outperforms  other  TMs.  ・LCTM  further  improves  performance  of  LCTM-­‐UNK.  à  LCTM  effecAvely  incorporates  OOV  words  in  held-­‐out  documents.    

Infer  topics  on  training  dataset

Predict  topic-­‐prop  of  held-­‐out  documents

Classify  by  topic  proporAon

Experimental  se\ng

Conclusion Introduced  LCTM  that  infers  topics  based  on  document-­‐level  co-­‐occurrence  of  latent  concepts.    Showed  that  LCTM  can  effecFvely  handle  OOV  words  in  held-­‐out  documents.    The  same  method  can  be  readily  applied  to  topic  models  that  extend  LDA.  

・

・

・

Figure 1: Projected latent concepts on the wordembedding space. Concept vectors are annotatedwith their representative concepts in parentheses.

words, we expect topically-related latent conceptsto co-occur many times, even in short texts withdiverse usage of words. This in turn promotestopic inference in LCTM.

LCTM further has the advantage of using con-tinuous word embedding. Traditional LDA as-sumes a fixed vocabulary of word types. Thismodeling assumption prevents LDA from han-dling out of vocabulary (OOV) words in held-outdocuments. On the other hands, since our topicmodel operates on the continuous vector space, itcan naturally handle OOV words once their vectorrepresentation is provided.

The main contributions of our paper are as fol-lows: We propose LCTM that infers topics viadocument-level co-occurrence patterns of latentconcepts, and derive a collapsed Gibbs samplerfor approximate inference. We show that LCTMcan accurately represent short texts by outperform-ing conventional topic models in a clustering task.By means of a classification task, we furthermoredemonstrate that LCTM achieves superior perfor-mance to other state-of-the-art topic models inhandling documents with a high degree of OOVwords.

The remainder of the paper is organized as fol-lows: related work is summarized in Section 2,while LCTM and its inference algorithm are pre-sented in Section 3. Experiments on the 20News-groups are presented in Section 4, and a conclu-sion is presented in Section 5.

2 Related Work

There have been a number of previous studies ontopic models that incorporate word embeddings.The closest model to LCTM is Gaussian LDA

(Das et al., 2015), which models each topic asa Gaussian distribution over the word embeddingspace. However, the assumption that topics areunimodal in the embedding space is not appropri-ate, since topically related words such as ‘neural’and ‘networks’ can occur distantly from each otherin the embedding space. Nguyen et al. (2015) pro-posed topic models that incorporate informationof word vectors in modeling topic-word distribu-tions. Similarly, Petterson et al. (Petterson et al.,2010) exploits external word features to improvethe Dirichlet prior of the topic-word distributions.However, both of the models cannot handle OOVwords, because they assume fixed word types.

Latent concepts in LCTM are closely relatedto ‘constraints’ in interactive topic models (ITM)(Hu et al., 2014). Both latent concepts and con-straints are designed to group conceptually simi-lar words using external knowledge in an attemptto aid topic inference. The difference lies in theirmodeling assumptions: latent concepts in LCTMare modeled as Gaussian distributions over theembedding space, while constraints in ITM aresets of conceptually similar words that are interac-tively identified by humans for each topic. Eachconstraint for each topic is then modeled as amultinomial distribution over the constrained setof words that were identified as mutually relatedby humans. In Section 4, we consider a variant ofITM, whose constraints are instead inferred usingexternal word embeddings.

As regards short texts, a well-known topicmodel is Biterm Topic Model (BTM) (Yan etal., 2013). BTM directly models the genera-tion of biterms (pairs of words) in the whole cor-pus. However, the assumption that pairs of co-occurring words should be assigned to the sametopic might be too strong (Chen et al., 2015).

3 Latent Concept Topic Model

3.1 Generative ModelThe primary difference between LCTM and theconventional topic models is that LCTM describesthe generative process of word vectors in docu-ments, rather than words themselves.

Suppose α and β are parameters for the Dirich-let priors and let vd,i denote the word embeddingfor a word type wd,i. The generative model forLCTM is as follows.

1. For each topic k

(a) Draw a topic concept distribution φk ∼Dirichlet(β).