Date post: | 25-Dec-2015 |
Category: |
Documents |
Upload: | eleanore-barrett |
View: | 214 times |
Download: | 1 times |
Statistical NLPWinter 2009
Lecture 17: Unsupervised Learning IV
Tree-structured learning
Roger Levy
[thanks to Jason Eisner and Mike Frank for some slides]
Inferring tree structure from raw text
• We’ve looked at a few types of unsupervised inference of linguistic structure:• Words from unsegmented utterances• Semantic word classes from word bags (=documents)• Syntactic word classes from word sequences
• But a hallmark of language is recursive structure• To a first approximation, this is trees• How might we learn tree structures from raw text?
Is this the right problem?
• Before proceeding…are we tackling the right problem?• We’ve already seen how the structures giving high
likelihood to raw text may not be the structures we want• Also, it’s fairly clear that kids don’t learn language
structure from linguistic input alone…
• Real learning is cross-modal
• So why study unsupervised grammar induction from raw text?
words: “blue rings”objects: rings, big bird
words: “and green rings”objects: rings, big bird
words: “and yellow rings”objects: rings, big bird
words: “Bigbird! Do you want to hold the rings?”
objects: big bird
Why induce grammars from raw text?
• I don’t think this is a trivial question to answer. But there are some answers:• Practical: if unsupervised grammar induction worked
well, it’d save a whole lot of annotation effort• Practical: we need tree structure to get compositional
semantics to work• Scientific: unsupervised grammar induction may help us
a place an upper bound on how much grammatical knowledge must be innate
• Theoretical: the models and algorithms that make unsupervised grammar induction work well may help us with other, related tasks (e.g., machine translation)
Plan of today
• We’ll start with a look at an early but influential clustering-based method (Clark, 2001)
• We’ll then talk about EM for PCFGs• Then we’ll talk about the next influential work (setting
the bar for all modern work), Klein & Manning (2004)
Distributional clustering for syntax
• How do we evaluate unsupervised grammar induction?
• Basically like we evaluate supervised parsing: through bracketing accuracy
• But this means that we don’t have to focus on inducing a grammar
• We can focus on inducing constituents in sentences instead
• This contrast has been called structure search versus parameter search
Distributional clustering for syntax (II)
• What makes a good constituent?• Old insight from Zellig Harris (Chomsky’s teacher):
something can appear in a consistent set of contextsSarah saw a dog standing near a tree.
Sarah saw a big dog standing near a tree.
Sarah saw me standing near a tree.
Sarah saw three posts standing near a tree.
• Therefore we could try to characterize the space of possible constituents by the contexts they appear in
• We could then do clustering in this space
Distributional clustering for syntax (III)
• Clark, 2001: k-means clustering of common tag sequences occurring in context distributions
• Applied to 12 million words from the British National Corpus
• The results were promising:
(determiner)
“the big dog”
“the very big dog”“the dog”“the dog near the tree”
Distributional clustering for syntax (IV)
• But…there was a problem:
• Clustering picks up both constituents and things that can’t be constituents (distituents)
“big dog the”
“dog big the”“dog the”“dog and big”
good at distinguishing constituent types bad at weeding out distituents
Distributional clustering for syntax (V)
• Clark 2001’s solution:“with real constituents, there is high mutual information
between the symbol occurring before the putative constituent and the symbol after…”
• Filter out distituents on the basis of low mutual information
• (controlling for distance between symbols)
constituents
distituents
Distributional clustering for syntax (VI)
• The final step: train an SCFG using vanilla maximum-likelihood estimation
• (hand-label the clusters to get interpretability)
Ten most frequent rules expanding “NP”
Structure vs. parameter search
• This did pretty well• But the clustering-based structure-search approach is
throwing out a hugely important type of information• (did you see what it is?)• …bracketing consistency!
Sarah gave the big dog the most enthuisiastic hug.
• Parameter-estimation approaches automatically incorporate this type of information
• This motivated the work of Klein & Manning (2002)
both can’t be a constituent at once
Interlude
• Before we discuss later work: how do we do parameter search for PCFGs?
• The EM algorithm is key• But as with HMM learning, there are exponentially
many structures to consider• We need dynamic programming (but a different sort
than for HMMs)
Review: EM for HMMs
• For the M-step, the fundamental quantity we need is the expected counts of state-to-state transitions
Probability of getting from beginning to state si at time t
Probability of getting from state sj at time t+1 to the end
Probability of transitioning from state sj at time t to state sj at time t+1, emitting ot
αi
βj
aijbijot
EM for PCFGs
• In PCFGs, for the M-step, the fundamental quantity is the expected counts of various rule rewrites
the familiar inside probability
something new: outside probability
Inside & outside probabilities
• In Manning & Schuetze (1999), here’s the picture:
You’re interested in node Nj
outside probability of Nj
inside probability of Nj
Compute Bottom-Up by CKY
VP(1,5) = p(flies like an arrow | VP) = …
S(0,5) = p(time flies like an arrow | S)
= NP(0,1) * VP(1,5) * p(S NP VP|S) + …
S
NPtime
VP
Vflies
PP
Plike
NP
Detan
N arrow
p( | S) = p(S NP VP | S) * p(NP time | NP)
* p(VP V PP | VP)
* p(V flies | V) * …
time 1 flies 2 like 3 an 4 arrow 5
0
NP3Vst3
NP10S8 S13
NP24NP24S22S27S27
1NP4VP4
NP18S21VP18
2P2V5
PP12VP16
3 Det1
NP10
4 N8
1 S NP VP6 S Vst NP2 S S PP
1 VP V NP2 VP VP PP
1 NP Det N2 NP NP PP3 NP NP NP
0 PP P NP
Compute Bottom-Up by CKY
time 1 flies 2 like 3 an 4 arrow 5
0
NP2-3
Vst2-3
NP 2-10 S 2-8
S 2-13
NP 2-24
NP 2-24
S 2-22
S 2-27
S 2-27
NP2-4
VP 2-4
NP 2-18
S 2-21
VP 2-18
2P2-2 V2-5
PP 2-12
VP 2-16
3 Det 2-1
NP 2-10
4 N2-8
2-1 S NP VP2-6 S Vst NP2-2 S S PP
2-1 VP V NP2-2 VP VP PP
2-1 NP Det N2-2 NP NP PP2-3 NP NP NP
2-0 PP P NP
Compute Bottom-Up by CKY
S 2-22
time 1 flies 2 like 3 an 4 arrow 5
0
NP2-3
Vst2-3
NP 2-10 S 2-8
S 2-13
NP 2-24
NP 2-24
S 2-22
S 2-27
S 2-27
S 2-22
NP2-4
VP 2-4
NP 2-18
S 2-21
VP 2-18
2P2-2 V2-5
PP 2-12
VP 2-16
3 Det 2-1
NP 2-10
4 N2-8
2-1 S NP VP2-6 S Vst NP2-2 S S PP
2-1 VP V NP2-2 VP VP PP
2-1 NP Det N2-2 NP NP PP2-3 NP NP NP
2-0 PP P NP
Compute Bottom-Up by CKY
S 2-27
time 1 flies 2 like 3 an 4 arrow 5
0
NP2-3
Vst2-3
NP 2-10 S 2-8
S 2-13
NP 2-24
NP 2-24
S 2-22
S 2-27
S 2-27
NP2-4
VP 2-4
NP 2-18
S 2-21
VP 2-18
2P2-2 V2-5
PP 2-12
VP 2-16
3 Det 2-1
NP 2-10
4 N2-8
2-1 S NP VP2-6 S Vst NP2-2 S S PP
2-1 VP V NP2-2 VP VP PP
2-1 NP Det N2-2 NP NP PP2-3 NP NP NP
2-0 PP P NP
The Efficient Version: Add as we go
time 1 flies 2 like 3 an 4 arrow 5
0
NP2-3
Vst2-3
NP 2-10 S 2-8
+2-13
NP 2-24
+2-24
S 2-22
+2-27
+2-27
NP2-4
VP 2-4
NP 2-18
S 2-21
VP 2-18
2P2-2 V2-5
PP 2-12
VP 2-16
3 Det 2-1
NP 2-10
4 N2-8
2-1 S NP VP2-6 S Vst NP2-2 S S PP
2-1 VP V NP2-2 VP VP PP
2-1 NP Det N2-2 NP NP PP2-3 NP NP NP
2-0 PP P NP
The Efficient Version: Add as we go
+2-22 +2-27
PP(2,5)
S(0,2) s(0,5)
s(0,2)* PP(2,5)*p(S S PP | S)
Compute probs bottom-up (CKY)
for width := 2 to n (* build smallest first *)
for i := 0 to n-width (* start *)
let k := i + width (* end *)
for j := i+1 to k-1 (* middle *)
for all grammar rules X Y Z
X(i,k) += p(X Y Z | X) * Y(i,j) * Z(j,k) X
Y Z
i j k
what if you changed + to max?
need some initialization up here for the width-1 case
VP(1,5) = p(flies like an arrow | VP)
VP(1,5) = p(time VP today | S)
Inside & Outside Probabilities S
NPtime
VP
VP NPtoday
Vflies
PP
Plike
NP
Detan
N arrow
VP(1,5) * VP(1,5) = p(time [VP flies like an arrow] today | S)
“inside” the VP
“outside” the VP
VP(1,5) = p(flies like an arrow | VP)
VP(1,5) = p(time VP today | S)
Inside & Outside Probabilities S
NPtime
VP
VP NPtoday
Vflies
PP
Plike
NP
Detan
N arrow
VP(1,5) * VP(1,5) = p(time flies like an arrow today & VP(1,5) | S)
/ S(0,6)
p(time flies like an arrow today | S)
= p(VP(1,5) | time flies like an arrow today, S)
VP(1,5) = p(flies like an arrow | VP)
VP(1,5) = p(time VP today | S)
Inside & Outside Probabilities S
NPtime
VP
VP NPtoday
Vflies
PP
Plike
NP
Detan
N arrow
So VP(1,5) * VP(1,5) / s(0,6) is probability that there is a VP here,given all of the observed data (words)
strictly analogousto forward-backwardin the finite-state case!
Start
C
H
C
H
C
H
V(1,2) = p(flies | V)
VP(1,5) = p(time VP today | S)
Inside & Outside Probabilities S
NPtime
VP
VP NPtoday
Vflies
PP
Plike
NP
Detan
N arrow
So VP(1,5) * V(1,2) * PP(2,5) / s(0,6) is probability that there is VP V PP here,given all of the observed data (words)
PP(2,5) = p(like an arrow | PP)
… or is it?
sum prob over all position triples like (1,2,5) toget expected c(VP V PP); reestimate PCFG!
V(1,2) = p(flies | V)
VP(1,5) = p(time VP today | S)
Inside & Outside Probabilities S
NPtime
VP
VP NPtoday
Vflies
PP
Plike
NP
Detan
N arrow
So VP(1,5) * p(VP V PP) * V(1,2) * PP(2,5) / s(0,6) is probability that there is VP V PP here (at 1-2-5),given all of the observed data (words)
PP(2,5) = p(like an arrow | PP)
Compute probs bottom-up
Vflies
PP
Plike
NP
Detan
N arrow
V(1,2) PP(2,5)
VPVP(1,5)
p(flies | V)
p(like an arrow | PP)
p(V PP | VP)+=
*
*
p(flies like an arrow | VP)
Summary: VP(1,5) += p(V PP | VP) * V(1,2) * PP(2,5)
inside 1,5 inside 1,2 inside 2,5
Compute probs top-down(uses probs as well)
Vflies
PP
Plike
NP
Detan
N arrow
PP(2,5)V(1,2)
VPVP(1,5)
VP
NPtoday
NPtime
+= p(time VP today | S)
* p(V PP | VP) *
p(like an arrow | PP)
p(time V like an arrow today | S)
S
Summary: V(1,2) += p(V PP | VP) * VP(1,5) * PP(2,5)
outside 1,2 outside 1,5 inside 2,5
Compute probs top-down(uses probs as well)
Vflies
PP
Plike
NP
Detan
N arrow
V(1,2)
PP(2,5)
VPVP(1,5)
VP
NPtoday
NPtime
+= p(time VP today | S)
* p(V PP | VP)
* p(flies| V)
p(time flies PP today | S)
S
Summary: PP(2,5) += p(V PP | VP) * VP(1,5) * V(1,2)
outside 2,5 outside 1,5 inside 1,2
Compute probs bottom-up
When you build VP(1,5), from VP(1,2) and VP(2,5) during CKY,increment VP(1,5) by
p(VP VP PP) * VP(1,2) * PP(2,5)
Why? VP(1,5) is total probability of all derivations p(flies like an arrow | VP)and we just found another.(See earlier slide of CKY chart.)
VPflies
PP
Plike
NP
Detan
N arrow
VP(1,2) PP(2,5)
VPVP(1,5)
Compute probs bottom-up (CKY)
for width := 2 to n (* build smallest first *)
for i := 0 to n-width (* start *)
let k := i + width (* end *)
for j := i+1 to k-1 (* middle *)
for all grammar rules X Y Z
X(i,k) += p(X Y Z) * Y(i,j) * Z(j,k) X
Y Z
i j k
Compute probs top-down (reverse CKY)
for width := 2 to n (* build smallest first *)
for i := 0 to n-width (* start *)
let k := i + width (* end *)
for j := i+1 to k-1 (* middle *)
for all grammar rules X Y Z
Y(i,j) += ???
Z(j,k) += ???
n downto 2 “unbuild” biggest first
X
Y Z
i j k
Compute probs top-down
After computing during CKY, revisit constits in reverse order (i.e., bigger constituents first).
When you “unbuild” VP(1,5) from VP(1,2) and VP(2,5), increment VP(1,2) by
VP(1,5) * p(VP VP PP) * PP(2,5)
and increment PP(2,5) by
VP(1,5) * p(VP VP PP) * VP(1,2)
S
NPtime
VP
VP NPtoday
VP(1,5)
VP(1,2) PP(2,5)
already computed onbottom-up pass
VP(1,2) is total prob of all ways to gen VP(1,2) and all outside words.
VP PP PP(2,5)VP(1,2)
Compute probs top-down (reverse CKY)
for width := 2 to n (* build smallest first *)
for i := 0 to n-width (* start *)
let k := i + width (* end *)
for j := i+1 to k-1 (* middle *)
for all grammar rules X Y Z
Y(i,j) += X(i,k) * p(X Y Z) * Z(j,k)
Z(j,k) += X(i,k) * p(X Y Z) * Y(i,j)
n downto 2 “unbuild” biggest first
X
Y Z
i j k
Back to the actual history
• Early approaches: Pereira & Schabes (1992), Carroll & Charniak (1994)• Initialize a simple PCFG using some “good guesses”• Use EM to re-estimate
• Horrible failure
EM on a constituent-context model
• Klein & Manning (2002) used a generative model that constructed sentences through bracketings• (1) choose a bracketing of a sentence• (2) for each span in the bracketing, (independently)
generate a yield and a context
• This is horribly deficient, but it worked well!
yieldcontext
Adding a dependency component
• Klein & Manning (2004) added a dependency component, which wasn’t deficient – a nice generative model!
• The dependency component introduced valence – the idea that a governor’s dependent-seeking behavior changes once it has a dependent
Constituent-context + dependency
• Some really nifty factorized-inference techniques allowed these models to be combined
• The results are very good
More recent developments
• Improved parameter search (Smith, 2006)• Bayesian methods (Liang et al., 2007)• Unsupervised all-subtrees parsing (Bod 2007)