Unsupervised Partial Parsing: Thesis defense

Unsupervised Partial Parsing

Elias Ponvert

Department of LinguisticsThe University of Texas at Austin

Dissertation DefenseJuly 27, 2011

Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 1 / 62

1 Goals and contributions

2 Unsupervised partial parsingMain resultsDiscussion

3 Cascaded parsingMain resultsDiscussion

4 Concluding remarks


Research goals

Generally:Develop computational models to learn humanlanguage

Hello!


Research goalsSpecifically:Learn to predict constituent structure from raw text

the cat saw the red dog run

⇓


Why unsupervised parsing?1 Less reliance on annotated training

Hello!

2 Apply to new languages and domains

Særær manannær man

mæþæn


Assumptions made in parser learning

S

NP VPPP

P

on

NP

N

Sunday

Det

the

A

brown

N

bear

V

sleeps

,

,

Getting these labels right AS WELL AS the structureof the tree is hard



P

on

N

Sunday

Det

the

A

brown

N

bear

V

sleeps

,

,

So the task is to identify the structure alone



on Sunday the brown bear

sleeps,

Learning operates from gold-standard parts-of-speech(POS) rather than raw text

P N Det A N

V,

on Sunday , the brown bear sleepsP N , Det A N V

Klein & Manning 2003 CCMBod 2006a, 2006bKlein & Manning 2005 DMVSuccessors to DMV: - Smith 2006, Smith & Cohen 2009, Headden et al 2009, Spitkovsky et al 2010ab, &c

J. Gao et al 2003, 2004Seginer 2007

this work


Unsupervised parsing: desiderata

Raw text

Standard NLP / extensible

Scalable and fast


Contributions

• Unsupervised parsing satisfying thesedesiderata is possible

• Unsupervised partial parsing: predicting localconstituents with high accuracy

• Cascaded models: building constituent structurebottom up


Outline






A new approach: start from the bottom

Unsupervised Partial Parsing =segmentation of (non-overlapping) multiword constituents


Unsupervised segmentation of constituentsleaves some room for interpretation

Possible segmentations• ( the cat ) in ( the hat ) knows ( a lot ) about that

• ( the cat ) ( in the hat ) knows ( a lot ) ( about that )

• ( the cat in the hat ) knows ( a lot about that )

• ( the cat in the hat ) ( knows a lot about that )

• ( the cat in the hat ) ( knows a lot ) ( about that )


Defining UPP by evaluation1. Constituent chunks:

non-hierarchical multiword constituentsS

NP

D

The

N

Cat

PP

P

in

NP

D

the

N

hat

VP

V

knows

NP

D

a

N

lot

PP

P

about

NP

N

that


Defining UPP by evaluation2. Base NPs:

non-recursive noun phrases

S

NP

D

The

N

Cat

PP

P

in

NP

D

the

N

hat

VP

V

knows

NP

D

a

N

lot

PP

P

about

NP

N

that


Multilingual data for direct evaluation

English WSJGerman NegraChinese CTB

Sentences Types TokensWSJ Penn Treebank 49K 44K 1M

Negra Negra German Corpus 21K 49K 300KCTB Penn Chinese Treebank 19K 37K 430K


Constituent chunks and NPs in the data

WSJChunks 203KNPs 172KChunks ∩ NPs 161K

NegraChunks 59KNPs 33KChunks ∩ NPs 23K

CTBChunks 92KNPs 56KChunks ∩ NPs 43K


The benchmark: CCL parser

the cat

saw

the red dog

run

the0 ��

cat0

��

1 ��saw

0 �� 0

��the

0 ��red

0 ��

0�� dog

0�� run

0��

Common Cover Links representation

Constituency tree

Seginer (2007 ACL; 2007 PhD UvA)


Hypothesis

Segmentation can be learned bygeneralizing on phrasal boundaries


UPP as a tagging problem

Bthe

Icat

Oin

Bthe

Ihat

the cat in the hat

B Beginning of a constituentI Inside a constituent

O Not inside a constituent


Learning from boundaries

Bthe

Icat

Oin

Bthe

Ihat

the cat in the hat

STOP

#STOP

#


Unsupervised learning tag model for UPP

B

the

I

cat

O

in the

I

hat

STOP

#

STOP

#

O

B

I

O

B

I

O

B

O



B

the

I

cat

O

in the

I

hat

STOP

#

STOP

#

O

B

I

O

B

I

O

B

O



B

the

I

cat

O

in the

I

hat

STOP

#

STOP

#

O

B

I

O

B

I

O

B

O



B

the

I

cat

O

in the

I

hat

STOP

#

STOP

#

O

B

I

O

B

I

O

B

O



B

the

I

cat

O

in the

I

hat

STOP

#

STOP

#

O

B

I

O

B

I

O

B

O



B

the

I

cat

O

in the

I

hat

STOP

#

STOP

#

O

B

I

O

B

I

O

B

O


Decoding the tag model for UPP

B

the

I

cat in the

I

hat

STOP

#STOP

#O B


Decoding the tag model for UPP

B

the

I

cat in the

I

hat

STOP

#STOP

#O B


Learning from punctuation

Bon

Isunday

Bthe

Ibrown

Ibear

STOP

#STOP

#

on sunday , the brown bear sleeps

STOP

,O

sleeps


UPP: Models

P( ) ≈ P( ) P( | )B

the

I

cat

O

in

B

the

I

hat

Hidden Markov Model

B I

theBtheB I

Probabilistic right linear grammar

P( ) = P( ) P( | )theB I B I

BI

OB

I

thecat

inthe

hat

B

Ithe

Learning: expectation maximization (EM) viaforward-backward (run to convergence)


UPP: Models

P( ) ≈ P( ) P( | )B

the

I

cat

O

in

B

the

I

hat

Hidden Markov Model

B I

theBtheB I


P( ) = P( ) P( | )theB I B I

BI

OB

I

thecat

inthe

hat

B

Ithe

Decoding: ViterbiSmoothing: additive smoothing on emissions


UPP: Constraints on sequences

Bthe

Icat

Oin

Bthe

Ihat

the cat in the hat

STOP

#STOP

#

STOP B

O I

1


UPP evaluation: Setup

• Evaluation by comparison to treebank data• Standard train / development / test splits• Precision and recall on matched constituents• Benchmark: CCL• Both get tokenization, punctuation,

sentence boundaries


UPP evaluation: Chunking (F-score)

0 10 20 30 40 50 60 70 80

CTB

Negra

WSJ

CCL∗ HMM Chunker PRLG Chunker

CCL non-hierarchical constituentsFirst-level parsing output


UPP evaluation: Base NPs (F-score)

0 10 20 30 40 50 60 70 80

CTB

Negra

WSJ

CCL∗ HMM Chunker PRLG Chunker

CCL non-hierarchical constituentsFirst-level parsing output


PRLG example output(the seeds) already are in (the script)

(little chance) that (shane longman) is goingto recoup today

it would have (severe implications) for(farmers ’ policy) holders

(thames ’s u.s. marketing agent)(donald taffner) is preparing to do just that

and all (the while) (the bonds) are in(the baby ’s diaper)

(mr. rustin) is (senior correspondent) in(the journal ’s london bureau)


UPP: Review

• Sequence models can generalize on indicatorsfor phrasal boundaries

• Leads to improved unsupervised segmentation• Learn to predict NPs with high accuracy

• (English and German especially)


Outline






Question

How do UPP models capturenoun phrase structure?


What UPP models learn

B 100 · P(w|B)the 21.0a 8.7to 6.5’s 2.8in 1.9mr. 1.8its 1.6of 1.4an 1.4and 1.4

I 100 · P(w|I)% 1.8million 1.6be 1.3company 0.9year 0.8market 0.7billion 0.6share 0.5new 0.5than 0.5

O 100 · P(w|O)

of 5.8and 4.0in 3.7that 2.2to 2.1for 2.0is 2.0it 1.7said 1.7on 1.5

HMM Emissions: WSJ



B 100 · P(w|B)der the 13.0die the 12.2den the 4.4und and 3.3im in 3.2das the 2.9des the 2.7dem the 2.4eine a 2.1ein a 2.0

I 100 · P(w|I)uhr o’clock 0.8juni June 0.6jahren years 0.4prozent percent 0.4mark currency 0.3stadt city 0.3000 0.3millionen millions 0.3jahre year 0.3frankfurter Frankfurt 0.3

O 100 · P(w|O)

in in 3.4und and 2.7mit with 1.7fur for 1.6auf on 1.5zu to 1.4von of 1.3sich oneself 1.3ist is 1.3nicht not 1.2

HMM Emissions: Negra



B 100 · P(w|B)的 de, of 14.3一 one 3.1和 and 1.1两 two 0.9这 this 0.8有 have 0.8经济 economy 0.7各 each 0.7全 all 0.7不 no 0.6

I 100 · P(w|I)的 de 3.9了 (perf. asp.) 2.2个 ge (measure) 1.5年 year 1.3说 say 1.0中 middle 0.9上 on, above 0.9人 person 0.7大 big 0.7国 country 0.6

O 100 · P(w|O)

在 at, in 3.4是 is 2.4中国 China 1.4也 also 1.2不 no 1.2对 pair 1.1和 and 1.0的 de 1.0将 fut. tns. 1.0有 have 1.0

HMM Emissions: CTB


Question

What about the PRLG, why does it do somuch better than the HMM?


Question

P( ) ≈ P( ) P( | )B

the

I

cat

O

in

B

the

I

hat

Hidden Markov Model

B I

theBtheB I


P( ) = P( ) P( | )theB I B I

BI

OB

I

thecat

inthe

hat

B

Ithe


What’s wrong with this picture?



O 100 · P(w|O)


• ’s occurs (immediately) before several terms thatappear after B


What’s wrong with this picture?



O 100 · P(w|O)


• ’s occurs (immediately) before several terms thatappear after B


PRLG rule probabilities

B 100 · P(B → w q)B → the I 28.2B → a I 11.7B → mr. I 2.4B → its I 2.2B → an I 1.9B → his I 1.0B → this I 1.0B → their I 1.0B → some I 0.7B → new I 0.6

I 100 · P(I → w q)I → ’s I 2.6I → and I 1.3I → % O 1.1I → million O 0.6I → new I 0.5I → million STOP 0.5I → company O 0.5I → year O 0.4I → & I 0.4I → million I 0.4

O 100 · P(O → w q)O → of B 3.8O → to O 3.6O → in B 2.5O → and O 1.7O → to B 1.7O → of O 1.6O → in O 1.5O → and B 1.4O → for B 1.3O → it O 1.3






















Learning curves: Base NPs

10 20 30 40K

20

40

60

80

sentences10 20 30 40K

2060

100

20

40

60

80

F-s

core

EM iter sentences

1

0 20 40 60 80 100

20

40

60

80

EM iter

PRLG chunking model: WSJ



5 10 15K1020304050

sentences 5 10 15K20

80140

20

40

F-s

core

EM iter sentences

1

0 50 100 1501020304050

EM iter

PRLG chunking model: Negra



5 10 15K0

10

20

30

sentences 510 15K

2060

100

10

20

30

F-s

core

EM iter sentences

1

0 20 40 60 80 1000

10

20

30

EM iter

PRLG chunking model: CTB


Question

How much can these models learn?


Against a supervised benchmark

∼4500 10K 20K 30K 40K

20

40

60

80

WSJ Sentences

Base

NPs

F-sc

oreSupervised PRLGUnsupervised PRLG



∼2200 5K 10K 15K

10

20

30

40

50

Negra Sentences

Base

NPs

F-sc

oreSupervised PRLGUnsupervised PRLG



5 10 15K

10

20

30

40

50

CTB Sentences

Base

NPs

F-sc

ore

Supervised PRLGUnsupervised PRLG


Negra/CTB training much smaller than WSJ

10K 20K 30K 40K

20

40

60

80

Negra PRLG

WSJ PRLG

CTB PRLG

Sentences

Base

NPs

F-sc

ore


Treebank precisionS

NP

D

The

N

Cat

PP

P

in

NP

D

the

N

hat

VP

V

knows

NP

D

a

N

lot

PP

P

about

NP

N

that

(the cat in the hat) knows (a lot) (about that)

• Constituent chunks: Prec = 2/3, Rec = 2/3, F = 2/3

• Base NPs: Prec = 1/3, Rec = 1/2

• Treebank precision: 3/3Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 38 / 62

On chunking the CTB

3 20 40 60 80

10

30

50

EM Iterations

Treebank precision

Base NPs F-scoreConstituent chunk F-score


Question.

Do these models scale?


Chunking with training from Gigaword NYT

+160K +320K +480K +640K

50

60

70

80

90

+NYT Sentences

Treebank precision

Base NPs F

Const. chunks F


Chunking with training from Gigaword NYT

WSJ +160K +320K +480K +640K

50

60

70

80

90

+NYT Sentences

Treebank precision

Base NPs F

Const. chunks F


Outline






Question

Are we limited to segmentation?


Hypothesis

Identification of higher level constituentscan also be learned by generalizing onphrasal boundaries


Cascaded UPP: 1 Segment raw text

there is no asbestos in our products now



Cascaded UPP: 2 Choose stand-ins for phrases

our productsis no asbestos


there in nowis our


Cascaded UPP: 3 Segment text + phrasal stand-ins

there in nowis our

there in nowis our


Cascaded UPP: 4 Choose stand-ins and repeat steps 3–4

our products

in

is no asbestos

there

there in nowis our

is in now


Cascaded UPP: 5 Unwind to output tree

our products

in

is no asbestos

there

is in now

thereis no asbestos in our products

now


Cascaded UPP: Review

• Separate models learned at each cascade level• Models share hyper-parameters (smoothing etc)• Choice of pseudowords as phrasal stand-ins• Pseudoword-identification: corpus frequency• Cascade run to convergence


Right-branching baseline

the quick brown fox jumped over the lazy dog

thequick

brownfox

jumpedover

thelazy dog


Right-branching baseline

a Lorillard spokeswoman said , this is an old story

a

Lorillard

spokeswoman said

this

is

an

old story


Cascaded UPP: Evaluation

0 10 20 30 40 50

CTB

Negra

WSJ

Constituents F-score

Baseline CCLCascaded HMM Cascaded PRLG


Another benchmark: CCM

Constituent-context model (Klein & Manning, 2002)

• Generative probabilistic model• Gold-standard POS• Short sentences


Evaluation on ≤10 word setences

0 10 20 30 40 50 60 70

CTB

Negra

WSJ

Constituents F-score

Baseline CCM CCLCascaded HMM Cascaded PRLG


Example parses

two share

a house almost devoid of furniture

Gold standardtwo

share

a housealmost devoid

offurniture

Cascaded PRLG – WSJ correctincorrect


Example parses

what

is one to think of all this

Gold standardwhat

is

one

to

think

of

all this

Cascaded PRLG – WSJ correctincorrect


Example parses

diethe

csuCSU

tutdoes

dasthis in

in

bayernBavaria

dochnevertheless

auchalso

sehrvery

erfolgreichsuccessfully

Nevertheless, the CSU does this in Bavaria very successfully as well

Gold standard

die csutut das

in bayerndoch auch

sehr erfolgreich

Cascaded PRLG – Negra correctincorrect


Example parses

beiwith

denthe

windsorsWindsors

bleibtstays

alleseverything

inin der

the

familiefamily

With the Windsors everything stays in the family.

Gold standard

bei den windsorsbleibt alles

in der familie



Example parses

immerever

mehrmore

anlagenteilemachine parts

uberalternover-age

(with) more and more machine parts over-age



Outline






Question

How do these cascaded chunkers work?


Recall of NPs and PPs

NPs PPsLev 1 Lev 2 Lev 1 Lev 2

WSJ PRLG 77.5 78.3 9.1 77.6Negra HMM 54.7 62.3 24.8 48.1CTB PRLG 30.9 33.6 31.6 47.1


Prec / Rec trade-offs in the cascade

1 2 3 4 5 6 720

40

60

80

Levels

Precision Recall F-score

1WSJ PRLG



1 2 3 4 5 6 7

30

40

50

Levels


1Negra PRLG



1 2 3 4 5 6 7

20304050

Levels


1CTB PRLG


Learning curves

10K 20K 30K 40K35

40

45

50

WSJ Sentences

F-sc

ore

PRLGCCL

HMM


Learning curves

5K 10K 15K25

30

35

40

Negra Sentences

F-sc

ore

PRLG

HMM

CCL


Learning curves

5K 10K 15K

20

30

40

CTB Sentences

F-sc

ore

PRLGHMM

CCL


Outline






What we’ve learned

• Unsupervised identification of base NPs andlocal constituents is possible

• A cascade of chunking models for raw textparsing has state-of-the-art results


Future directions

• Improvements to the sequence models• Better phrasal stand-in (pseudoword)

construction• Learning joint models rather than a cascade


Historical note

First known computational natural language parserTransformations and Discourse Analysis ProjectZellig Harris & colleagues, UPenn 1950s - 1960s


Historical note

To the best of our knowledge, this is the firstapplication of FSTs to parsing. The programconsisted of the following phases:

1. Dictionary look-up.2. Replacement of some ‘grammatical idioms’ by a

single part of speech.3. Rule based part of speech disambiguation.4. A right to left FST composed with a left to right

FST for computing ‘simple noun phrases’.

Joshi & Hopely 1997


Historical note

To the best of our knowledge, this is the firstapplication of FSTs to parsing. The programconsisted of the following phases:

4. A left to right FST for computing ‘simpleadjuncts’ such as prepositional phrases andadverbial phrases.

5. A left to right FST for computing simple verbclusters.

6. A left to right ‘FST’ for computing clauses.

Joshi & Hopely 1997


Thanks!


Date post:	08-May-2015
Category:	Technology
Upload:	elias-ponvert
View:	621 times
Download:	3 times

Unsupervised Partial Parsing: Thesis defense

Technology