Type-Based MCMCpliang/papers/type-naacl2010-talk.pdf · Type-Based MCMC NAACL 2010 { Los Angeles...

Type-Based MCMC

NAACL 2010 – Los Angeles

Percy Liang Michael I. Jordan Dan Klein

Type-Based MCMC

NAACL 2010 – Los Angeles

Percy Liang Michael I. Jordan Dan Klein

Learning latent-variable models

· · · DT NNS VBD · · ·

· · · some stocks rose · · ·

· · · NN NNS VBD · · ·

· · · tech stocks fell · · ·

· · · DT NNS VBD · · ·

· · · the stocks soared · · ·

· · · WRB NNS VBD · · ·

· · · how stocks dipped · · ·

· · · DT NNS VBD · · ·

· · · many stocks created · · ·

2


· · · DT NNS VBD · · ·


· · · NN NNS VBD · · ·


· · · DT NNS VBD · · ·


· · · WRB NNS VBD · · ·


· · · DT NNS VBD · · ·


2


θ

· · · DT NNS VBD · · ·


· · · NN NNS VBD · · ·


· · · DT NNS VBD · · ·


· · · WRB NNS VBD · · ·


· · · DT NNS VBD · · ·


2


· · · DT NNS VBD · · ·


· · · NN NNS VBD · · ·


· · · DT NNS VBD · · ·


· · · WRB NNS VBD · · ·


· · · DT NNS VBD · · ·


2


· · · DT NNS VBD · · ·


· · · NN NNS VBD · · ·


· · · DT NNS VBD · · ·


· · · WRB NNS VBD · · ·


· · · DT NNS VBD · · ·


Variables are highly dependent

2


· · · DT NNS VBD · · ·


· · · NN NNS VBD · · ·


· · · DT NNS VBD · · ·


· · · WRB NNS VBD · · ·


· · · DT NNS VBD · · ·



Simple: token-based (e.g., Gibbs sampler)

2


· · · DT NNS VBD · · ·


· · · NN NNS VBD · · ·


· · · DT NNS VBD · · ·


· · · WRB NNS VBD · · ·


· · · DT NNS VBD · · ·



Simple: token-based (e.g., Gibbs sampler)⇒ local optima, slow mixing

2


· · · DT NNS VBD · · ·


· · · NN NNS VBD · · ·


· · · DT NNS VBD · · ·


· · · WRB NNS VBD · · ·


· · · DT NNS VBD · · ·




Classic: sentence-based (e.g., EM)

2


· · · DT NNS VBD · · ·


· · · NN NNS VBD · · ·


· · · DT NNS VBD · · ·


· · · WRB NNS VBD · · ·


· · · DT NNS VBD · · ·





Structural dependencies

2


· · · DT NNS VBD · · ·


· · · NN NNS VBD · · ·


· · · DT NNS VBD · · ·


· · · WRB NNS VBD · · ·


· · · DT NNS VBD · · ·






dynamic programming

2


· · · DT NNS VBD · · ·


· · · NN NNS VBD · · ·


· · · DT NNS VBD · · ·


· · · WRB NNS VBD · · ·


· · · DT NNS VBD · · ·






dynamic programming

New: type-based

2


· · · DT NNS VBD · · ·


· · · NN NNS VBD · · ·


· · · DT NNS VBD · · ·


· · · WRB NNS VBD · · ·


· · · DT NNS VBD · · ·






dynamic programming

New: type-based

Parameter dependencies

2


· · · DT NNS VBD · · ·


· · · NN NNS VBD · · ·


· · · DT NNS VBD · · ·


· · · WRB NNS VBD · · ·


· · · DT NNS VBD · · ·






dynamic programming

New: type-based


exchangeability

2


Dependencies between adjacent variables

3



prob

abili

ty

NN VBZ NN NNworker demands meeting resistance

3



prob

abili

ty


NN NNS VBG NNworker demands meeting resistance

3



prob

abili

ty



Token-based: update only one variable at a time

3



prob

abili

ty



NN VBZ VBG NNworker demands meeting resistance


3



prob

abili

ty





3



prob

abili

ty




Token-based: update only one variable at a timeProblem: need to go downhill before uphill

3



Sentence-based: update all variables in a sentence

prob

abili

ty




Token-based: update only one variable at a timeProblem: need to go downhill before uphill

3


Dependencies between variables with shared parameters

4



prob

abili

ty

VBZ VBD ...stocks rose ...VBZ VBD ...stocks fell ...VBZ VBD ...stocks took ...VBZ VBD ...stocks shot ...

4



prob

abili

ty


NNS VBD ...stocks rose ...NNS VBD ...stocks fell ...NNS VBD ...stocks took ...NNS VBD ...stocks shot ...

4



prob

abili

ty



NNS VBD ...stocks rose ...VBZ VBD ...stocks fell ...VBZ VBD ...stocks took ...VBZ VBD ...stocks shot ...

Token-based:

4



prob

abili

ty




NNS VBD ...stocks rose ...NNS VBD ...stocks fell ...VBZ VBD ...stocks took ...VBZ VBD ...stocks shot ...

Token-based:

4



prob

abili

ty





NNS VBD ...stocks rose ...NNS VBD ...stocks fell ...NNS VBD ...stocks took ...VBZ VBD ...stocks shot ...

Token-based:

4



prob

abili

ty






Token-based: need to go downhill a lot before going uphill

4



Type-based: update all variables of one type

prob

abili

ty







4




prob

abili

ty







1. Parameter dependencies create deeper valleys

4




prob

abili

ty







1. Parameter dependencies create deeper valleys

2. Sentence-based cannot handle these dependencies

4

What exactly is a type?

How can we update all variables of a type efficiently?

5

Formal setup

Parameters θ: vector of conditional probabilities

6

Formal setup


θT:V,N: from state V, probability of transitioning to state N

6

Formal setup



p(θ) is product of Dirichlets [prior]

6

Formal setup




Choices z: specifies values of latent and observed variables

6

Formal setup




Choices z: specifies values of latent and observed variables8 9

· · · V N · · ·stocks

6

Formal setup





z ∈ z[T :V,N at 8]

8 9

· · · V N · · ·stocks

6

Formal setup





z ∈ z j(z) [parameter used][T :V,N at 8] T:V,N

8 9

· · · V N · · ·stocks

6

Formal setup






8 9

· · · V N · · ·stocks

p(z | θ) =∏

z∈z θj(z) [likelihood]

6

Formal setup






8 9

· · · V N · · ·stocks

p(z | θ) =∏


p(z) =∫p(z | θ)p(θ)dθ [marginal likelihood]

6

Formal setup






8 9

· · · V N · · ·stocks

p(z | θ) =∏



Observations x: observed part of z (e.g., the words)

6

Formal setup






8 9

· · · V N · · ·stocks

p(z | θ) =∏



Observations x: observed part of z (e.g., the words)

Goal: sample from p(z | x) [not p(z | x,θ)]6

Exchangeability

Sufficient statistics n: # times parameters were used in z

7

Exchangeability

Sufficient statistics n: # times parameters were used in znT:V,N: # times that state V transitioned to state N

7

Exchangeability


Rewrite likelihood:

p(z | θ) =∏

j θnj

j

7

Exchangeability


Rewrite likelihood:

p(z | θ) =∏

j θnj

j = simple function of n and θ

7

Exchangeability


Rewrite likelihood:

p(z | θ) =∏

j θnj


p(z) = simple function of n

7

Exchangeability


Rewrite likelihood:

p(z | θ) =∏

j θnj


p(z) = simple function of n (key: exchangeability)

7

Exchangeability


Rewrite likelihood:

p(z | θ) =∏

j θnj



z1

· · · D V V · · ·many stocks rose

· · · D N V · · ·tech stocks were

7

Exchangeability


Rewrite likelihood:

p(z | θ) =∏

j θnj



z1 z2



· · · D N V · · ·many stocks rose

· · · D V V · · ·tech stocks were

7

Exchangeability


Rewrite likelihood:

p(z | θ) =∏

j θnj



z1 z2





7

Exchangeability


Rewrite likelihood:

p(z | θ) =∏

j θnj



z1 z2





z1 and z2 have same sufficient statistics

7

Exchangeability


Rewrite likelihood:

p(z | θ) =∏

j θnj



z1 z2





z1 and z2 have same sufficient statistics ⇒ p(z1) = p(z2)

7

Types

Type of a variable: its dependent parameter components

8

Types


For HMM, type = assignment to Markov blanket

8

Types



· · · D V · · ·many stocks rose

8

Types



type( ) = (D, stocks, V)


8

Types





· · · D V · · ·tech stocks were

· · · D V · · ·the stocks have

· · · D V · · ·how stocks from

8

Types






· · · D N V · · ·the stocks have

· · · D N V · · ·how stocks from

Assignments[VVNN]

8

Types






· · · D V V · · ·the stocks have


Assignments[VVNN][VNVN]

8

Types







· · · D V V · · ·how stocks from

Assignments[VVNN][VNVN][VNNV]

8

Types








Assignments[VVNN][VNVN][VNNV][NVVN]

8

Types








Assignments[VVNN][VNVN][VNNV][NVVN][NVNV]

8

Types








Assignments[VVNN][VNVN][VNNV][NVVN][NVNV][NNVV]

8

Types








Assignments[VVNN][VNVN][VNNV][NVVN][NVNV][NNVV]

p(assignment) only depends on number of Vs and Ns8

Sampling same-type variables

Goal: sample an assignment of a set of same-type variables

9



m = 0 m = 1 m = 2 m = 3 m = 4p0 [VVVV] p1 [VVVN]

p1 [VVNV]p1 [VNVV]p1 [NVVV]

p2 [VVNN]p2 [VNVN]p2 [VNNV]p2 [NVVN]p2 [NVNV]p2 [NNVV]

p3 [VNNN]p3 [NVNN]p3 [NNVN]p3 [NNNV]

p4 [NNNN]

9



Algorithm:

1. Choose m ∈ {0, . . . , B} with prob. ∝(

Bm

)pm





p4 [NNNN]

9



Algorithm:

1. Choose m ∈ {0, . . . , B} with prob. ∝(

Bm

)pm

2. Choose assignment uniformly from column m





p4 [NNNN]

9

Full algorithm

Iterate: · · · D V V · · ·many stocks rose




10

Full algorithm

Iterate:1. Choose a position

e.g., 2





10

Full algorithm


e.g., 2

2. Add variables with same typee.g., (D, stocks, V)





10

Full algorithm


e.g., 2


3. Sample me.g., 1 ⇒ {V, V, V, N}


· · · D V · · ·tech stocks were

· · · D V · · ·the stocks have

· · · D V · · ·how stocks from

10

Full algorithm


e.g., 2


3. Sample me.g., 1 ⇒ {V, V, V, N}

4. Sample assignmente.g., [VNVV]





10

Experimental setup: sampling algorithms

2

tech

5

stocks

3

rose

1

in

8

heavy

2

trading

4

worker

6

demands

8

meeting

1

resistance

2

many

4

stocks

3

shot

8

up

6

today

5

investors

2

await

4

stocks

3

news

11


2

tech

5

stocks

3

rose

1

in

8

heavy

2

trading

4

worker

6

demands

8

meeting

1

resistance

2

many

4

stocks

3

shot

8

up

6

today

5

investors

2

await

4

stocks

3

news

Token-based sampler (Token)1. Choose token

11


2

tech

5

stocks

3

rose

1

in

8

heavy

2

trading

4

worker

6

demands

7

meeting

1

resistance

2

many

4

stocks

3

shot

8

up

6

today

5

investors

2

await

4

stocks

3

news


2. Update conditioned on rest

11


2

tech

5

stocks

3

rose

1

in

8

heavy

2

trading

4

worker

6

demands

8

meeting

1

resistance

2

many

4

stocks

3

shot

8

up

6

today

5

investors

2

await

4

stocks

3

news



Sentence-based sampler (Sentence)1. Choose sentence

11


3

tech

4

stocks

6

rose

1

in

2

heavy

7

trading

4

worker

6

demands

8

meeting

1

resistance

2

many

4

stocks

3

shot

8

up

6

today

5

investors

2

await

4

stocks

3

news





11


2

tech

5

stocks

3

rose

1

in

8

heavy

2

trading

4

worker

6

demands

8

meeting

1

resistance

2

many

4

stocks

3

shot

8

up

6

today

5

investors

2

await

4

stocks

3

news





Type-based sampler (Type)1. Choose type

11


2

tech

4

stocks

3

rose

1

in

8

heavy

2

trading

4

worker

6

demands

8

meeting

1

resistance

2

many

3

stocks

3

shot

8

up

6

today

5

investors

2

await

4

stocks

3

news





Type-based sampler (Type)1. Choose type


11

Experimental setup: models/tasks/datasets

Hidden Markov model (HMM)

12




Part-of-speech induction

12





12





Dataset: WSJ(49K sentences, 45 tags)

12






Unigram segmentation model (USM) [Goldwater et al., 2006]

12







l o o k a t t h e b o o k

Word segmentation

12








Word segmentation

12








Word segmentation

Dataset: CHILDES(9.7K sentences)

12








Word segmentation


Probabilistic tree-substitution grammar (PTSG) [Cohn et al., 2009]

12








Word segmentation



the

DT

sun

NN

NP

has

VBD

risen

VBN

VP

S

12








Word segmentation



the

DT

sun

NN

NP

has

VBD

risen

VBN

VP

S

12








Word segmentation



the

DT

sun

NN

NP

has

VBD

risen

VBN

VP

S


12

Token versus Type

HMM USM PTSG

13

Token versus Type

HMM USM PTSG

3 6 9 12

time (hr.)

-7.3e6

-6.9e6

-6.5e6

log-

likel

ihood

Token

13

Token versus Type

HMM USM PTSG

3 6 9 12

time (hr.)

-7.3e6

-6.9e6

-6.5e6

log-

likel

ihood

Token

Type

13

Token versus Type

HMM USM PTSG

3 6 9 12

time (hr.)

-7.3e6

-6.9e6

-6.5e6

log-

likel

ihood

2 4 6 8

time (min.)

-2.1e5

-1.9e5

-1.7e5

log-

likel

ihood

Token

Type

13

Token versus Type

HMM USM PTSG

3 6 9 12

time (hr.)

-7.3e6

-6.9e6

-6.5e6

log-

likel

ihood

2 4 6 8

time (min.)

-2.1e5

-1.9e5

-1.7e5

log-

likel

ihood

3 6 9 12

time (hr.)

-6.1e6

-5.9e6

-5.7e6

-5.5e6

-5.3e6

log-

likel

ihood

Token

Type

13

Token versus Type

HMM USM PTSG

3 6 9 12

time (hr.)

-7.3e6

-6.9e6

-6.5e6

log-

likel

ihood

2 4 6 8

time (min.)

-2.1e5

-1.9e5

-1.7e5

log-

likel

ihood

3 6 9 12

time (hr.)

-6.1e6

-5.9e6

-5.7e6

-5.5e6

-5.3e6

log-

likel

ihood

3 6 9 12

time (hr.)

35

40

45

50

55

60

tag

accu

racy

2 4 6 8

time (min.)

35

40

45

50

55

60

wor

dto

ken

F1

Token

Type

13

Sensitivity to initialization

14


(use few params.)η = 0

14



V V V Vworker demands meeting resistance

14





14



(use many params.)η = 1



14





P N D Vworker demands meeting resistance


14





P N D Vworker demands meeting resistance

l o o k a t t h e b o o k l o o k a t t h e b o o k

14


HMM USM PTSG

15


HMM USM PTSG

0.2 0.4 0.6 0.8 1.0

η

-7.2e6

-7.0e6

-6.8e6

-6.6e6

log-

likel

ihood

0.2 0.4 0.6 0.8 1.0

η

10

20

30

40

50

60

tag

accu

racy

Token

15


HMM USM PTSG

0.2 0.4 0.6 0.8 1.0

η

-7.2e6

-7.0e6

-6.8e6

-6.6e6

log-

likel

ihood

0.2 0.4 0.6 0.8 1.0

η

10

20

30

40

50

60

tag

accu

racy

Token

Type

15


HMM USM PTSG

0.2 0.4 0.6 0.8 1.0

η

-7.2e6

-7.0e6

-6.8e6

-6.6e6

log-

likel

ihood

0.2 0.4 0.6 0.8 1.0

η

-3.4e5

-3.1e5

-2.7e5

-2.3e5

-1.9e5

log-

likel

ihood

0.2 0.4 0.6 0.8 1.0

η

10

20

30

40

50

60

tag

accu

racy

0.2 0.4 0.6 0.8 1.0

η

10

20

30

40

50

60

wor

dto

ken

F1

Token

Type

15


HMM USM PTSG

0.2 0.4 0.6 0.8 1.0

η

-7.2e6

-7.0e6

-6.8e6

-6.6e6

log-

likel

ihood

0.2 0.4 0.6 0.8 1.0

η

-3.4e5

-3.1e5

-2.7e5

-2.3e5

-1.9e5

log-

likel

ihood

0.2 0.4 0.6 0.8 1.0

η

-5.7e6

-5.6e6

-5.5e6

-5.4e6

log-

likel

ihood

0.2 0.4 0.6 0.8 1.0

η

10

20

30

40

50

60

tag

accu

racy

0.2 0.4 0.6 0.8 1.0

η

10

20

30

40

50

60

wor

dto

ken

F1

Token

Type

15


HMM USM PTSG

0.2 0.4 0.6 0.8 1.0

η

-7.2e6

-7.0e6

-6.8e6

-6.6e6

log-

likel

ihood

0.2 0.4 0.6 0.8 1.0

η

-3.4e5

-3.1e5

-2.7e5

-2.3e5

-1.9e5

log-

likel

ihood

0.2 0.4 0.6 0.8 1.0

η

-5.7e6

-5.6e6

-5.5e6

-5.4e6

log-

likel

ihood

0.2 0.4 0.6 0.8 1.0

η

10

20

30

40

50

60

tag

accu

racy

0.2 0.4 0.6 0.8 1.0

η

10

20

30

40

50

60

wor

dto

ken

F1

Token

Type

Type less sensitive than Token

15

Alternatives

Can we get the gains of Type via simpler means?

16

Alternatives


• Annealing (Tokenanneal)

– Use p(z | x)1/T , temperature T from 5 to 1

17

Annealing

HMM USM PTSG

18

Annealing

HMM USM PTSG

3 6 9 12

time (hr.)

-7.3e6

-6.9e6

-6.5e6

log-

likel

ihood

3 6 9 12

time (hr.)

35

40

45

50

55

60

tag

accu

racy

Token

Type

18

Annealing

HMM USM PTSG

3 6 9 12

time (hr.)

-7.3e6

-6.9e6

-6.5e6

log-

likel

ihood

3 6 9 12

time (hr.)

35

40

45

50

55

60

tag

accu

racy

Token

Tokenanneal

Type

[anneal. hurts]

18

Annealing

HMM USM PTSG

3 6 9 12

time (hr.)

-7.3e6

-6.9e6

-6.5e6

log-

likel

ihood

2 4 6 8

time (min.)

-2.1e5

-1.9e5

-1.7e5

log-

likel

ihood

3 6 9 12

time (hr.)

35

40

45

50

55

60

tag

accu

racy

2 4 6 8

time (min.)

35

40

45

50

55

60

wor

dto

ken

F1

Token

Tokenanneal

Type

[anneal. hurts] [anneal. helps]

18

Annealing

HMM USM PTSG

3 6 9 12

time (hr.)

-7.3e6

-6.9e6

-6.5e6

log-

likel

iho

od

2 4 6 8

time (min.)

-2.1e5

-1.9e5

-1.7e5

log-

likel

iho

od

3 6 9 12

time (hr.)

-6.1e6

-5.9e6

-5.7e6

-5.5e6

-5.3e6

log-

likel

iho

od

3 6 9 12

time (hr.)

35

40

45

50

55

60

tag

accu

racy

2 4 6 8

time (min.)

35

40

45

50

55

60

wor

dto

ken

F1

Token

Tokenanneal

Type

[anneal. hurts] [anneal. helps] [no effect]

18

Alternatives




– More (random) mobility through space, but insufficient

19

Alternatives





• Sentence-based (Sentence)

20

Sentence

USM:

2 4 6 8

time (min.)

-2.1e5

-1.9e5

-1.7e5

log-

likel

ihood

2 4 6 8

time (min.)

35

40

45

50

55

60

wor

dto

ken

F1

Token

Type

21

Sentence

USM:

2 4 6 8

time (min.)

-2.1e5

-1.9e5

-1.7e5

log-

likel

ihood

2 4 6 8

time (min.)

35

40

45

50

55

60

wor

dto

ken

F1

Token

Type

Sentence

21

Sentence

USM:

2 4 6 8

time (min.)

-2.1e5

-1.9e5

-1.7e5

log-

likel

ihood

2 4 6 8

time (min.)

35

40

45

50

55

60

wor

dto

ken

F1

Token

Type

Sentence

Sentence performs comparably to Token,

but worse than Type

21

Sentence

USM:

2 4 6 8

time (min.)

-2.1e5

-1.9e5

-1.7e5

log-

likel

ihood

2 4 6 8

time (min.)

35

40

45

50

55

60

wor

dto

ken

F1

Token

Type

Sentence

Sentence performs comparably to Token,

but worse than Type

Sentence requires dynamic programming,

computationally more expensive than Token and Type

21

Alternatives






– Sentence handles structural dependencies

– Type handles parameter dependencies

22

Alternatives






– Sentence handles structural dependencies

– Type handles parameter dependencies

– Intuition: parameter dependencies more important inunsupervised learning

23

Summary and outlook

General strategy: update many dependent variables tractably

24

Summary and outlook


Variables sharing parameters very dependent

Type-based sampler updates exactly these variables

24

Summary and outlook




Techniques for tractability:

Old: exploit dynamic programming structure

24

Summary and outlook





Old: exploit dynamic programming structureNew: think about exchangeability

24

Summary and outlook






Tokens versus types:

24

Summary and outlook







• Older work operate on types (e.g., model merging)

24

Summary and outlook







• Older work operate on types (e.g., model merging)Larger updates, but greedy and brittle

24

Summary and outlook








• Recent methods operate on sentences or tokensSmaller updates, but softer and more robust

24

Summary and outlook








• Recent methods operate on sentences or tokensSmaller updates, but softer and more robust

Type-based sampling combines advantages of both

24

Thank you!

25

Date post:	18-Jan-2021
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

Type-Based MCMCpliang/papers/type-naacl2010-talk.pdf · Type-Based MCMC NAACL 2010 { Los Angeles...

Documents